---
title: "The local / self-hosted LLM stack"
type: "stack"
slug: "local-llm"
canonical_url: "https://www.graphcanon.com/stacks/local-llm"
---

# The local / self-hosted LLM stack

Running open models yourself for privacy, cost, or control. The stack is a serving runtime, optional fine-tuning, and supporting tooling.

## 1. Inference & Serving

Inference runtime - serve open-weight models efficiently on your hardware.

**When not to use:** Self-hosting rarely beats a hosted API on cost until you have steady, high-volume traffic.

**Compare:**
- [Ollama vs llama.cpp](/compare/ggml-org-llama-cpp-vs-ollama-ollama.md)

**Category:** [Inference & Serving](/categories/inference-serving.md)

- [transformers](/tools/huggingface-transformers.md) - Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models (★ 164,121) [Very active]
- [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 148,875) [Very active]
- [ollama](/tools/ollama-ollama.md) - Get up and running with various large language models using Ollama. (★ 177,524) [Very active]
- [vllm](/tools/vllm-project-vllm.md) - A high-throughput and memory-efficient inference and serving engine for LLMs (★ 87,847) [Very active]

## 2. Model Training

Fine-tuning - adapt an open model to your domain with LoRA or full fine-tunes.

**When not to use:** Try prompting and RAG first; fine-tuning is the answer to style/format, not missing knowledge.

**Category:** [Model Training](/categories/model-training.md)

- [transformers](/tools/huggingface-transformers.md) - Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models (★ 164,121) [Very active]
- [tensorflow](/tools/tensorflow-tensorflow.md) - An Open Source Machine Learning Framework for Everyone (★ 196,758) [Very active]
- [LLMs-from-scratch](/tools/rasbt-llms-from-scratch.md) - Implement a ChatGPT-like LLM in PyTorch from scratch, step by step (★ 102,733) [Very active]
- [keras](/tools/keras-team-keras.md) - Deep Learning for humans (★ 64,217) [Very active]

## 3. Developer Tools

Tooling & gateways - a unified API in front of your local models.

**When not to use:** Direct calls are fine for a single local model with one consumer.

**Category:** [Developer Tools](/categories/developer-tools.md)

- [ECC](/tools/affaan-m-ecc.md) - The agent harness performance optimization system for AI agents (★ 240,297) [Very active]
- [prompts.chat](/tools/f-prompts-chat.md) - Share, discover, and collect prompts from the community (★ 167,212) [Very active]
- [OpenHands](/tools/openhands-openhands.md) - AI-Driven Development (★ 84,154) [Very active]
- [Flowise](/tools/flowiseai-flowise.md) - Build AI Agents, Visually (★ 55,246) [Very active]

## Common questions

### What is the the local / self-hosted llm stack?

Running open models yourself for privacy, cost, or control. The stack is a serving runtime, optional fine-tuning, and supporting tooling.

### When should I use the the local / self-hosted llm stack?

Use this stack when your constraints match its layers: Inference & Serving, Model Training, Developer Tools. Each step on the page includes when-not-to-use guidance so you do not over-engineer.

### How does GraphCanon pick tools for the local / self-hosted llm stack?

Stack pages are editorial workflows over canonical categories. Representative tools are pulled live from the graph at render time (top adoption in each category), not a fixed marketing list.

### Where can I compare tools in this stack?

Follow category hubs and head-to-head compare pages linked from each step. Featured comparisons cover the highest-intent pairs (frameworks, vector DBs, agent runtimes). Start at [compare hub](/compare).

### Is there a machine-readable version of The local / self-hosted LLM stack?

Yes. Append `.md` to [`/stacks/local-llm`](/stacks/local-llm) or fetch [`/md/stacks/local-llm`](/md/stacks/local-llm) for a markdown twin with steps, when-not notes, and tool links.

---

**Machine-readable endpoints**

- JSON: [`/api/graphcanon/categories`](/api/graphcanon/categories)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)

_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_