---
title: "The local / self-hosted LLM stack"
type: "stack"
slug: "local-llm"
canonical_url: "https://www.graphcanon.com/stacks/local-llm"
---

# The local / self-hosted LLM stack

Running open models yourself for privacy, cost, or control. The stack is a serving runtime, optional fine-tuning, and supporting tooling.

## 1. Inference & Serving

Inference runtime - serve open-weight models efficiently on your hardware.

**When not to use:** Self-hosting rarely beats a hosted API on cost until you have steady, high-volume traffic.

**Compare:**
- [Ollama vs llama.cpp](/compare/ggml-org-llama-cpp-vs-ollama-ollama.md)

**Category:** [Inference & Serving](/categories/inference-serving.md)

- [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: state-of-the-art machine learning models for text, vision, audio, and multimodal models (★ 166,279) [Very active]
- [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (★ 152,445) [Very active]
- [ollama](/tools/ollama-ollama.md) - Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. (★ 181,196) [Very active]
- [vllm](/tools/vllm-project-vllm.md) - A high-throughput and memory-efficient inference and serving engine for LLMs (★ 92,075) [Very active]

## 2. Model Training

Fine-tuning - adapt an open model to your domain with LoRA or full fine-tunes.

**When not to use:** Try prompting and RAG first; fine-tuning is the answer to style/format, not missing knowledge.

**Category:** [Model Training](/categories/model-training.md)

- [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: state-of-the-art machine learning models for text, vision, audio, and multimodal models (★ 166,279) [Very active]
- [tensorflow](/tools/tensorflow-tensorflow.md) - An Open Source Machine Learning Framework for Everyone (★ 200,493) [Very active]
- [LLMs-from-scratch](/tools/rasbt-llms-from-scratch.md) - Implement a ChatGPT-like LLM in PyTorch from scratch, step by step (★ 105,176) [Very active]
- [caveman](/tools/juliusbrussee-caveman.md) - Reduces token usage by 65% through minimalistic language for coding agents (★ 106,401) [Very active]

## 3. Developer Tools

Tooling & gateways - a unified API in front of your local models.

**When not to use:** Direct calls are fine for a single local model with one consumer.

**Category:** [Developer Tools](/categories/developer-tools.md)

- [ECC](/tools/affaan-m-ecc.md) - Agent harness performance optimization system for AI agents and LLMs (★ 261,432) [Very active]
- [AutoGPT](/tools/significant-gravitas-autogpt.md) - Accessible AI for everyone, providing tools for focus on what matters (★ 187,424) [Very active]
- [dify](/tools/langgenius-dify.md) - Build Agentic workflows and RAG pipelines with rich AI model and tool support on one collaborative workspace. (★ 156,243) [Very active]
- [graphify](/tools/graphify-labs-graphify.md) - Transforms codebases into a queryable knowledge graph (★ 119,168) [Very active]

## Common questions

### What is the the local / self-hosted llm stack?

Running open models yourself for privacy, cost, or control. The stack is a serving runtime, optional fine-tuning, and supporting tooling.

### When should I use the the local / self-hosted llm stack?

Use this stack when your constraints match its layers: Inference & Serving, Model Training, Developer Tools. Each step on the page includes when-not-to-use guidance so you do not over-engineer.

### How does GraphCanon pick tools for the local / self-hosted llm stack?

Stack pages are editorial workflows over canonical categories. Representative tools are pulled live from the graph at render time (top adoption in each category), not a fixed marketing list.

### Where can I compare tools in this stack?

Follow category hubs and head-to-head compare pages linked from each step. Featured comparisons cover the highest-intent pairs (frameworks, vector DBs, agent runtimes). Start at [compare hub](/compare).

### Is there a machine-readable version of The local / self-hosted LLM stack?

Yes. Append `.md` to [`/stacks/local-llm`](/stacks/local-llm) or fetch [`/md/stacks/local-llm`](/md/stacks/local-llm) for a markdown twin with steps, when-not notes, and tool links.

---

**Machine-readable endpoints**

- JSON: [`/api/graphcanon/categories`](/api/graphcanon/categories)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)

_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_