--- title: "Inference & Serving" type: "category" slug: "inference-serving" canonical_url: "https://www.graphcanon.com/categories/inference-serving" tool_count: 412 --- # Inference & Serving *GraphCanon updated Oct 12, 2026* Model inference, serving, and local runtimes β€” deploying and running models efficiently (Ollama, vLLM, llama.cpp, TGI). 412 tools in this category (showing the top 60 by stars). ## Featured comparisons - [Yi vs Qwen](/compare/01-ai-yi-vs-qwenlm-qwen.md) - [Yi vs LLMs-from-scratch](/compare/01-ai-yi-vs-rasbt-llms-from-scratch.md) - [rig vs LocalAI](/compare/0xplaygrounds-rig-vs-mudler-localai.md) - [1Panel vs open-webui](/compare/1panel-dev-1panel-vs-open-webui-open-webui.md) - [CV vs self-llm](/compare/accumulatemore-cv-vs-datawhalechina-self-llm.md) - [gateway vs llmfit](/compare/adaline-gateway-vs-alexsjones-llmfit.md) - [gateway vs litellm](/compare/adaline-gateway-vs-berriai-litellm.md) - [gateway vs 9router](/compare/adaline-gateway-vs-decolua-9router.md) ## Stacks - [The local / self-hosted LLM stack](/stacks/local-llm.md) ## Tools - [transformers](/tools/huggingface-transformers.md) - πŸ€— Transformers: state-of-the-art machine learning models for text, vision, audio, and multimodal models (β˜… 166,279) [Very active] - [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (β˜… 152,445) [Very active] - [ollama](/tools/ollama-ollama.md) - Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. (β˜… 181,196) [Very active] - [vllm](/tools/vllm-project-vllm.md) - A high-throughput and memory-efficient inference and serving engine for LLMs (β˜… 92,075) [Very active] - [OmniRoute](/tools/diegosouzapw-omniroute.md) - Free AI gateway with multi-provider support and token savings (β˜… 68,316) [Very active] - [litellm](/tools/berriai-litellm.md) - The fastest, lightest AI Gateway (β˜… 59,051) [Very active] - [headroom](/tools/headroomlabs-ai-headroom.md) - Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. (β˜… 72,850) [Very active] - [anything-llm](/tools/mintplex-labs-anything-llm.md) - Self-hosted AI agent experience (β˜… 66,167) [Very active] - [LibreChat](/tools/danny-avila-librechat.md) - Enhanced ChatGPT Clone with extensive features and integrations for self-hosting (β˜… 44,416) [Very active] - [ray](/tools/ray-project-ray.md) - Unified framework for scaling AI and Python applications (β˜… 43,863) [Very active] - [LocalAI](/tools/mudler-localai.md) - Open-source AI engine for running various models on any hardware (β˜… 49,150) [Very active] - [sglang](/tools/sgl-project-sglang.md) - SGLang is a high-performance serving framework for large language models and multimodal models. (β˜… 36,123) [Very active] - [whisper.cpp](/tools/ggml-org-whisper-cpp.md) - C++ port of OpenAI's Whisper model for speech-to-text (β˜… 53,742) [Very active] - [onyx](/tools/onyx-dot-app-onyx.md) - Open Source AI Platform - AI Chat with advanced features that works with every LLM (β˜… 32,153) [Very active] - [mlflow](/tools/mlflow-mlflow.md) - Open source AI engineering platform for agents, LLMs, and ML models (β˜… 28,010) [Very active] - [ruflo](/tools/ruvnet-ruflo.md) - The leading agent meta-harness for intelligent multi-player swarms and autonomous workflows (β˜… 72,875) [Very active] - [netron](/tools/lutzroeder-netron.md) - Visualizer for neural network, deep learning and machine learning models (β˜… 33,524) [Very active] - [claude-mem](/tools/thedotmack-claude-mem.md) - Persistent Context Across Sessions for Every Agent (β˜… 91,018) [Very active] - [ncnn](/tools/tencent-ncnn.md) - High-performance neural network inference framework optimized for mobile platforms (β˜… 23,905) [Very active] - [airllm](/tools/lyogavin-airllm.md) - Inference for AirLLM 70B model with single 4GB GPU (β˜… 34,504) [Very active] - [screenpipe](/tools/screenpipe-screenpipe.md) - AI that records and analyzes everything you do, say, hear locally (β˜… 21,636) [Very active] - [generative-ai](/tools/googlecloudplatform-generative-ai.md) - Sample code and notebooks for Generative AI on Google Cloud, with Gemini Enterprise Agent Platform (β˜… 17,726) [Very active] - [openvino](/tools/openvinotoolkit-openvino.md) - OpenVINO is an open source toolkit for optimizing and deploying AI inference. (β˜… 10,874) [Very active] - [omlx](/tools/jundot-omlx.md) - LLM inference server with continuous batching and SSD caching for Apple Silicon (β˜… 21,934) [Very active] - [quivr](/tools/quivrhq-quivr.md) - Opiniated RAG for integrating GenAI in your apps (β˜… 39,533) [Active] - [pytorch](/tools/pytorch-pytorch.md) - Tensors and Dynamic neural networks in Python with strong GPU acceleration (β˜… 103,384) [Very active] - [txtai](/tools/neuml-txtai.md) - All-in-one AI framework for semantic search, LLM orchestration and language model workflows (β˜… 12,959) [Very active] - [khoj](/tools/khoj-ai-khoj.md) - Self-hostable AI second brain for personalized research and automation (β˜… 37,399) [Steady] - [dynamo](/tools/ai-dynamo-dynamo.md) - A Datacenter Scale Distributed Inference Serving Framework (β˜… 8,121) [Very active] - [inference](/tools/xorbitsai-inference.md) - Unified production-ready inference API for various LLMs and models (β˜… 9,575) [Very active] - [litgpt](/tools/lightning-ai-litgpt.md) - High-performance LLMs for pretraining, finetuning, and deployment (β˜… 13,667) [Very active] - [awesome-LLM-resources](/tools/wangrongsheng-awesome-llm-resources.md) - Summary of the world's best LLM resources. (β˜… 8,968) [Very active] - [oumi](/tools/oumi-ai-oumi.md) - Easily fine-tune, evaluate and deploy open source LLMs/VLMs (β˜… 9,387) [Very active] - [runanywhere-sdks](/tools/runanywhereai-runanywhere-sdks.md) - Production ready toolkit to run AI locally (β˜… 10,297) [Very active] - [ChuanhuChatGPT](/tools/gaizhenbiao-chuanhuchatgpt.md) - GUI for ChatGPT and LLMs with features like agents and file-based QA. (β˜… 15,272) [Very active] - [ai-guide](/tools/liyupi-ai-guide.md) - ε…θ΄ΉεΌ€ζ”Ύηš„AIηŸ₯识共享平台 (β˜… 18,766) [Active] - [llama.cpp](/tools/ggml-org-llama-cpp.md) - LLM inference in C/C++ (β˜… 128,639) [Very active] - [ml-engineering](/tools/stas00-ml-engineering.md) - Machine Learning Engineering Open Book (β˜… 19,009) [Very active] - [TensorRT-LLM](/tools/nvidia-tensorrt-llm.md) - Python API for defining and optimizing Large Language Models for efficient inference on NVIDIA GPUs (β˜… 14,645) [Very active] - [metaflow](/tools/netflix-metaflow.md) - Build, Manage and Deploy AI/ML Systems (β˜… 10,273) [Very active] - [llamafile](/tools/mozilla-ai-llamafile.md) - Distribute and run LLMs with a single file. (β˜… 25,999) [Very active] - [OpenRLHF](/tools/openrlhf-openrlhf.md) - Scalable and High-performance Agentic RL Framework based on Ray (β˜… 10,014) [Very active] - [DeepSpeed](/tools/deepspeedai-deepspeed.md) - Deep learning optimization library for efficient distributed training and inference (β˜… 43,065) [Very active] - [bifrost](/tools/maximhq-bifrost.md) - Fast Enterprise AI Gateway with Adaptive Load Balancer and Guardrails (β˜… 8,179) [Very active] - [llm-action](/tools/liguodongiot-llm-action.md) - Aims to share large model technology principles and practical experience (large model engineering, application implementation) (β˜… 24,898) [Active] - [OpenLLM](/tools/bentoml-openllm.md) - Run any open-source LLMs as OpenAI compatible API endpoint in the cloud. (β˜… 12,527) [Very active] - [mcp](/tools/awslabs-mcp.md) - Open source MCP Servers for AWS (β˜… 9,711) [Very active] - [llm-app](/tools/pathwaycom-llm-app.md) - Ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with live data (β˜… 58,920) [Steady] - [LMFlow](/tools/optimalscale-lmflow.md) - An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (β˜… 8,487) [Steady] - [awesome-free-llm-apis](/tools/mnfst-awesome-free-llm-apis.md) - List of Permanent Free LLM API (β˜… 7,852) [Active] - [SillyTavern](/tools/sillytavern-sillytavern.md) - LLM Frontend for Power Users (β˜… 33,508) [Very active] - [BentoML](/tools/bentoml-bentoml.md) - The easiest way to serve AI apps and models (β˜… 8,847) [Active] - [CosyVoice](/tools/funaudiollm-cosyvoice.md) - Multi-lingual large voice generation model with full-stack abilities for inference, training and deployment. (β˜… 23,695) [Slowing] - [pytorch-lightning](/tools/lightning-ai-pytorch-lightning.md) - Pretrain, finetune ANY AI model of ANY size on 1 or 10,000+ GPUs with zero code changes. (β˜… 31,363) [Very active] - [llmfit](/tools/alexsjones-llmfit.md) - Hardware-aware model recommendation tool (β˜… 36,747) [Very active] - [MNN](/tools/alibaba-mnn.md) - Blazing-fast, lightweight inference engine for high-performance on-device LLMs and Edge AI (β˜… 16,039) [Very active] - [lmdeploy](/tools/internlm-lmdeploy.md) - Toolkit for compressing, deploying, and serving LLMs (β˜… 8,044) [Very active] - [go-stock](/tools/arvinlovegood-go-stock.md) - AI-empowered stock analysis tool for multiple markets (β˜… 7,568) [Very active] - [llm-universe](/tools/datawhalechina-llm-universe.md) - ι’ε‘ε°η™½εΌ€ε‘θ€…ηš„LLM应用开发教程 (β˜… 13,803) [Active] - [mlops-zoomcamp](/tools/datatalksclub-mlops-zoomcamp.md) - Free MLOps course from DataTalks.Club (β˜… 15,308) [Very active] ## Common questions ### What are the best inference & serving tools? GraphCanon ranks Inference & Serving tools by GitHub adoption and freshness. transformers is the current leader (166,279 stars). See the full list on this page - sorted by stars, with [maintenance labels](/glossary/trust-and-signals/maintenance-label) and graph relationships. ### How does GraphCanon rank Inference & Serving tools? We sort by GitHub stars and push recency on category pages, not paid placement. Alternatives and compare pages use [typed graph edges](/glossary/knowledge-graph/typed-edge) (alternative, successor, integrates_with) plus shared categories - constraint-first, not marketing votes. ### How many tools are in Inference & Serving? 412 published tools are tagged with Inference & Serving in the GraphCanon knowledge graph. ### What are popular Inference & Serving comparisons? Head-to-head compare pages in this category include Yi vs Qwen, Yi vs LLMs-from-scratch, rig vs LocalAI. Each comparison uses live GitHub stats and optional [trust signals](/glossary/trust-and-signals/trust-signal) - see the comparisons block on this page. ### Which stacks use Inference & Serving? Curated workflow pages that include Inference & Serving: [The local / self-hosted LLM stack](/stacks/local-llm). Each stack step includes when-not-to-use guidance. ### Is there a machine-readable Inference & Serving list? Yes. Append `.md` to this URL or fetch [`/md/categories/inference-serving`](/md/categories/inference-serving) for a markdown twin. The JSON API exposes the same corpus at [`/api/graphcanon/categories/inference-serving`](/api/graphcanon/categories/inference-serving). --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/categories/inference-serving`](/api/graphcanon/categories/inference-serving) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_