The local / self-hosted LLM stack
Running open models yourself for privacy, cost, or control. The stack is a serving runtime, optional fine-tuning, and supporting tooling.
GraphCanon updated Oct 4, 2026
86views this month
Inference runtime - serve open-weight models efficiently on your hardware.
When not to use: Self-hosting rarely beats a hosted API on cost until you have steady, high-volume traffic.
transformers
huggingface/transformers
🤗 Transformers: state-of-the-art machine learning models for text, vision, audio, and multimodal models
open-webui
open-webui/open-webui
User-friendly AI Interface
ollama
ollama/ollama
Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
vllm
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
Fine-tuning - adapt an open model to your domain with LoRA or full fine-tunes.
When not to use: Try prompting and RAG first; fine-tuning is the answer to style/format, not missing knowledge.
transformers
huggingface/transformers
🤗 Transformers: state-of-the-art machine learning models for text, vision, audio, and multimodal models
tensorflow
tensorflow/tensorflow
An Open Source Machine Learning Framework for Everyone
LLMs-from-scratch
rasbt/LLMs-from-scratch
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
caveman
JuliusBrussee/caveman
Reduces token usage by 65% through minimalistic language for coding agents
Tooling & gateways - a unified API in front of your local models.
When not to use: Direct calls are fine for a single local model with one consumer.
ECC
affaan-m/ECC
Agent harness performance optimization system for AI agents and LLMs
AutoGPT
Significant-Gravitas/AutoGPT
Accessible AI for everyone, providing tools for focus on what matters
dify
langgenius/dify
Build Agentic workflows and RAG pipelines with rich AI model and tool support on one collaborative workspace.
graphify
Graphify-Labs/graphify
Transforms codebases into a queryable knowledge graph
Common questions
- What is the the local / self-hosted llm stack?
- Running open models yourself for privacy, cost, or control. The stack is a serving runtime, optional fine-tuning, and supporting tooling.
- When should I use the the local / self-hosted llm stack?
- Use this stack when your constraints match its layers: Inference & Serving, Model Training, Developer Tools. Each step on the page includes when-not-to-use guidance so you do not over-engineer.
- How does GraphCanon pick tools for the local / self-hosted llm stack?
- Stack pages are editorial workflows over canonical categories. Representative tools are pulled live from the graph at render time (top adoption in each category), not a fixed marketing list.
- Where can I compare tools in this stack?
- Follow category hubs and head-to-head compare pages linked from each step. Featured comparisons cover the highest-intent pairs (frameworks, vector DBs, agent runtimes). Start at compare hub.
- Is there a machine-readable version of The local / self-hosted LLM stack?
- Yes. Append
.mdto `/stacks/local-llm` or fetch `/md/stacks/local-llm` for a markdown twin with steps, when-not notes, and tool links.