The local / self-hosted LLM stack
Running open models yourself for privacy, cost, or control. The stack is a serving runtime, optional fine-tuning, and supporting tooling.
GraphCanon updated 5d · 55 views this month
Inference runtime - serve open-weight models efficiently on your hardware.
When not to use: Self-hosting rarely beats a hosted API on cost until you have steady, high-volume traffic.
transformers
huggingface/transformers
Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models
open-webui
open-webui/open-webui
User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
ollama
ollama/ollama
Get up and running with various large language models using Ollama.
vllm
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
Fine-tuning - adapt an open model to your domain with LoRA or full fine-tunes.
When not to use: Try prompting and RAG first; fine-tuning is the answer to style/format, not missing knowledge.
transformers
huggingface/transformers
Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models
tensorflow
tensorflow/tensorflow
An Open Source Machine Learning Framework for Everyone
LLMs-from-scratch
rasbt/LLMs-from-scratch
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
keras
keras-team/keras
Deep Learning for humans
Tooling & gateways - a unified API in front of your local models.
When not to use: Direct calls are fine for a single local model with one consumer.
ECC
affaan-m/ECC
The agent harness performance optimization system for AI agents
prompts.chat
f/prompts.chat
Share, discover, and collect prompts from the community
OpenHands
OpenHands/OpenHands
AI-Driven Development
Flowise
FlowiseAI/Flowise
Build AI Agents, Visually
Common questions
- What is the the local / self-hosted llm stack?
- Running open models yourself for privacy, cost, or control. The stack is a serving runtime, optional fine-tuning, and supporting tooling.
- When should I use the the local / self-hosted llm stack?
- Use this stack when your constraints match its layers: Inference & Serving, Model Training, Developer Tools. Each step on the page includes when-not-to-use guidance so you do not over-engineer.
- How does GraphCanon pick tools for the local / self-hosted llm stack?
- Stack pages are editorial workflows over canonical categories. Representative tools are pulled live from the graph at render time (top adoption in each category), not a fixed marketing list.
- Where can I compare tools in this stack?
- Follow category hubs and head-to-head compare pages linked from each step. Featured comparisons cover the highest-intent pairs (frameworks, vector DBs, agent runtimes). Start at compare hub.
- Is there a machine-readable version of The local / self-hosted LLM stack?
- Yes. Append
.mdto `/stacks/local-llm` or fetch `/md/stacks/local-llm` for a markdown twin with steps, when-not notes, and tool links.