{"data":{"node":{"slug":"ggml-org-llama-cpp","name":"llama.cpp","tagline":"LLM inference in C/C++","github_url":"https://github.com/ggml-org/llama.cpp","owner":"ggml-org","repo":"llama.cpp","owner_avatar_url":"https://avatars.githubusercontent.com/u/134263123?v=4","primary_language":"C++","stars":128639,"forks":23359,"topics":["ggml"],"archived":false,"github_pushed_at":"2026-09-18T08:13:37+00:00","maintenance_label":"Very active","stars_delta_30d":5698,"url":"https://www.graphcanon.com/tools/ggml-org-llama-cpp","markdown_url":"https://www.graphcanon.com/tools/ggml-org-llama-cpp.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ggml-org-llama-cpp","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ggml-org-llama-cpp"},"categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"c","name":"c++"},{"slug":"ggml","name":"ggml"},{"slug":"llm-inference","name":"llm-inference"}],"edges":[{"type":"related","direction":"out","explanation":null,"successor_context":null,"tool":{"slug":"unslothai-unsloth","name":"unsloth","tagline":"Local UI for running and training LLMs and diffusion models","github_url":"https://github.com/unslothai/unsloth","owner":"unslothai","repo":"unsloth","owner_avatar_url":"https://avatars.githubusercontent.com/u/150920049?v=4","primary_language":"Python","stars":76344,"forks":6962,"topics":["agent","ai","chatgpt","deepseek","fine-tuning","gemma","image-generation","llama","llm","llms","openai","python","qwen","reinforcement-learning","self-hosted","stable-diffusion","text-to-speech","tts","ui","unsloth"],"archived":false,"github_pushed_at":"2026-09-18T08:00:38+00:00","maintenance_label":"Very active","stars_delta_30d":6723,"url":"https://www.graphcanon.com/tools/unslothai-unsloth","markdown_url":"https://www.graphcanon.com/tools/unslothai-unsloth.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/unslothai-unsloth","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=unslothai-unsloth"}},{"type":"alternative","direction":"out","explanation":"Both airllm and llama.cpp offer lightweight GPU inference options for large language models, differing mainly in their implementation and optimization approaches.","successor_context":null,"tool":{"slug":"lyogavin-airllm","name":"airllm","tagline":"Inference for AirLLM 70B model with single 4GB GPU","github_url":"https://github.com/lyogavin/airllm","owner":"lyogavin","repo":"airllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/1113905?v=4","primary_language":"Jupyter Notebook","stars":34504,"forks":3626,"topics":["chinese-llm","chinese-nlp","finetune","generative-ai","instruct-gpt","instruction-set","llama","llm","lora","open-models","open-source","open-source-models","qlora"],"archived":false,"github_pushed_at":"2026-09-18T06:25:07+00:00","maintenance_label":"Very active","stars_delta_30d":10321,"url":"https://www.graphcanon.com/tools/lyogavin-airllm","markdown_url":"https://www.graphcanon.com/tools/lyogavin-airllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/lyogavin-airllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=lyogavin-airllm"}},{"type":"integrates_with","direction":"out","explanation":"`wllama` provides WebAssembly bindings for `llama.cpp`, enabling on-browser LLM inference, indicating they integrate well.","successor_context":null,"tool":{"slug":"ngxson-wllama","name":"wllama","tagline":"WebAssembly binding for llama.cpp - Enabling on-browser LLM inference","github_url":"https://github.com/ngxson/wllama","owner":"ngxson","repo":"wllama","owner_avatar_url":"https://avatars.githubusercontent.com/u/7702203?v=4","primary_language":"TypeScript","stars":1196,"forks":121,"topics":["llama","llamacpp","llm","wasm","webassembly"],"archived":false,"github_pushed_at":"2026-09-06T12:17:08+00:00","maintenance_label":"Very active","stars_delta_30d":37,"url":"https://www.graphcanon.com/tools/ngxson-wllama","markdown_url":"https://www.graphcanon.com/tools/ngxson-wllama.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ngxson-wllama","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ngxson-wllama"}},{"type":"alternative","direction":"out","explanation":"Both `tiny-vllm` and `llama.cpp` are designed to provide high-performance LLM inference engines in C++, making them alternatives.","successor_context":null,"tool":{"slug":"jmaczan-tiny-vllm","name":"tiny-vllm","tagline":"Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM","github_url":"https://github.com/jmaczan/tiny-vllm","owner":"jmaczan","repo":"tiny-vllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/18054202?v=4","primary_language":"C++","stars":1120,"forks":92,"topics":["ai","attention","batching","course","cpp","cuda","hpc","inference","llm","llm-inference","pagedattention","tiny-vllm","vllm"],"archived":false,"github_pushed_at":"2026-09-15T19:46:09+00:00","maintenance_label":"Very active","stars_delta_30d":173,"url":"https://www.graphcanon.com/tools/jmaczan-tiny-vllm","markdown_url":"https://www.graphcanon.com/tools/jmaczan-tiny-vllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/jmaczan-tiny-vllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=jmaczan-tiny-vllm"}},{"type":"related","direction":"out","explanation":"`Awesome-LLM-Inference` is a curated list of inference resources and papers which may include mentions or resources related to `llama.cpp`, making them adjacent yet not directly connected.","successor_context":null,"tool":{"slug":"xlite-dev-awesome-llm-inference","name":"Awesome-LLM-Inference","tagline":"A curated list of LLM/VLM inference papers with codes","github_url":"https://github.com/xlite-dev/Awesome-LLM-Inference","owner":"xlite-dev","repo":"Awesome-LLM-Inference","owner_avatar_url":"https://avatars.githubusercontent.com/u/204302598?v=4","primary_language":"Python","stars":5508,"forks":435,"topics":["awesome-llm","deepseek","deepseek-r1","deepseek-v3","flash-attention","flash-attention-3","flash-mla","llm-inference","minimax-01","mla","paged-attention","qwen3","tensorrt-llm","vllm"],"archived":false,"github_pushed_at":"2026-08-14T12:23:49+00:00","maintenance_label":"Steady","stars_delta_30d":93,"url":"https://www.graphcanon.com/tools/xlite-dev-awesome-llm-inference","markdown_url":"https://www.graphcanon.com/tools/xlite-dev-awesome-llm-inference.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/xlite-dev-awesome-llm-inference","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=xlite-dev-awesome-llm-inference"}},{"type":"integrates_with","direction":"out","explanation":"`TurboLLM` can use `llama.cpp` as one of its engines for running LLM models, showing integration potential.","successor_context":null,"tool":{"slug":"mohitsoni48-turbollm","name":"TurboLLM","tagline":"Run any local LLM engine auto-tuned to your GPU with polished web UI and OpenAI/Anthropic-compatible API","github_url":"https://github.com/mohitsoni48/TurboLLM","owner":"mohitsoni48","repo":"TurboLLM","owner_avatar_url":"https://avatars.githubusercontent.com/u/63787789?v=4","primary_language":"TypeScript","stars":274,"forks":38,"topics":["ai","anthropic-api","claude-code","gguf","gpu","inference","llama-cpp","llama-server","llm","local-llm","offline","openai-api","self-hosted"],"archived":false,"github_pushed_at":"2026-09-19T13:11:45+00:00","maintenance_label":"Very active","stars_delta_30d":49,"url":"https://www.graphcanon.com/tools/mohitsoni48-turbollm","markdown_url":"https://www.graphcanon.com/tools/mohitsoni48-turbollm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/mohitsoni48-turbollm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=mohitsoni48-turbollm"}},{"type":"alternative","direction":"out","explanation":"`VLLM` and `llama.cpp` both offer high-throughput and efficient LLM inference engines, thus they are considered alternatives.","successor_context":null,"tool":{"slug":"vllm-project-vllm","name":"vllm","tagline":"A high-throughput and memory-efficient inference and serving engine for LLMs","github_url":"https://github.com/vllm-project/vllm","owner":"vllm-project","repo":"vllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/136984999?v=4","primary_language":"Python","stars":92075,"forks":22363,"topics":["amd","blackwell","cuda","deepseek","deepseek-v3","gpt","gpt-oss","inference","kimi","llama","llm","llm-serving","model-serving","moe","openai","pytorch","qwen","qwen3","tpu","transformer"],"archived":false,"github_pushed_at":"2026-09-18T07:28:54+00:00","maintenance_label":"Very active","stars_delta_30d":4228,"url":"https://www.graphcanon.com/tools/vllm-project-vllm","markdown_url":"https://www.graphcanon.com/tools/vllm-project-vllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/vllm-project-vllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=vllm-project-vllm"}},{"type":"integrates_with","direction":"out","explanation":"`ggrun` requires `llama.cpp` to function as the engine for launching GGUF models, indicating a dependency.","successor_context":null,"tool":{"slug":"raketenkater-ggrun","name":"ggrun","tagline":"Auto-tuned launcher for GGUF models on llama.cpp with OpenAI-compatible server","github_url":"https://github.com/raketenkater/ggrun","owner":"raketenkater","repo":"ggrun","owner_avatar_url":"https://avatars.githubusercontent.com/u/49783786?v=4","primary_language":"Go","stars":275,"forks":18,"topics":["cuda","gguf","golang","inference-server","llama-cpp","llamacpp","llm","local-llm","localllama","metal","moe","multi-gpu","ollama-alternative","openai-api","self-hosted","speculative-decoding","vulkan"],"archived":false,"github_pushed_at":"2026-09-19T09:42:32+00:00","maintenance_label":"Very active","stars_delta_30d":11,"url":"https://www.graphcanon.com/tools/raketenkater-ggrun","markdown_url":"https://www.graphcanon.com/tools/raketenkater-ggrun.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/raketenkater-ggrun","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=raketenkater-ggrun"}},{"type":"alternative","direction":"in","explanation":"Both LocalAI and llama.cpp provide means to run inference on large language models, with LocalAI offering a more versatile runtime.","successor_context":null,"tool":{"slug":"mudler-localai","name":"LocalAI","tagline":"Open-source AI engine for running various models on any hardware","github_url":"https://github.com/mudler/LocalAI","owner":"mudler","repo":"LocalAI","owner_avatar_url":"https://avatars.githubusercontent.com/u/2420543?v=4","primary_language":"Go","stars":49150,"forks":4457,"topics":["agents","ai","api","audio-generation","decentralized","distributed","image-generation","libp2p","llama","llm","mamba","mcp","musicgen","object-detection","rerank","stable-diffusion","text-generation","tts"],"archived":false,"github_pushed_at":"2026-09-18T08:36:38+00:00","maintenance_label":"Very active","stars_delta_30d":650,"url":"https://www.graphcanon.com/tools/mudler-localai","markdown_url":"https://www.graphcanon.com/tools/mudler-localai.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/mudler-localai","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=mudler-localai"}},{"type":"depends_on","direction":"in","explanation":"Jan runs LLMs offline, which likely uses llama.cpp under the hood for inference in C/C++.","successor_context":null,"tool":{"slug":"janhq-jan","name":"jan","tagline":"Jan is an open source alternative to ChatGPT that runs 100% offline on your computer.","github_url":"https://github.com/janhq/jan","owner":"janhq","repo":"jan","owner_avatar_url":"https://avatars.githubusercontent.com/u/102363196?v=4","primary_language":"TypeScript","stars":44518,"forks":3034,"topics":["chatgpt","gpt","llamacpp","llm","localai","open-source","self-hosted","tauri"],"archived":false,"github_pushed_at":"2026-09-18T08:22:25+00:00","maintenance_label":"Very active","stars_delta_30d":498,"url":"https://www.graphcanon.com/tools/janhq-jan","markdown_url":"https://www.graphcanon.com/tools/janhq-jan.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/janhq-jan","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=janhq-jan"}},{"type":"alternative","direction":"in","explanation":"Both MLC-LLM and llama.cpp are focused on LLM inference but with different hardware support and optimizations.","successor_context":null,"tool":{"slug":"mlc-ai-mlc-llm","name":"mlc-llm","tagline":"Universal LLM Deployment Engine with ML Compilation","github_url":"https://github.com/mlc-ai/mlc-llm","owner":"mlc-ai","repo":"mlc-llm","owner_avatar_url":"https://avatars.githubusercontent.com/u/106173866?v=4","primary_language":"Python","stars":23063,"forks":2111,"topics":["language-model","llm","machine-learning-compilation","tvm"],"archived":false,"github_pushed_at":"2026-07-31T03:03:18+00:00","maintenance_label":"Active","stars_delta_30d":103,"url":"https://www.graphcanon.com/tools/mlc-ai-mlc-llm","markdown_url":"https://www.graphcanon.com/tools/mlc-ai-mlc-llm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/mlc-ai-mlc-llm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=mlc-ai-mlc-llm"}},{"type":"depends_on","direction":"in","explanation":"LEANN can potentially depend on llama.cpp for efficient LLM inference in C/C++, which aligns with its aim of providing fast and accurate RAG applications locally.","successor_context":null,"tool":{"slug":"startrail-org-leann","name":"LEANN","tagline":"RAG on Everything with LEANN","github_url":"https://github.com/StarTrail-org/LEANN","owner":"StarTrail-org","repo":"LEANN","owner_avatar_url":"https://avatars.githubusercontent.com/u/288858980?v=4","primary_language":"Python","stars":12785,"forks":1145,"topics":["ai","faiss","gpt-oss","langchain","llama-index","llm","localstorage","offline-first","ollama","privacy","python","rag","retrieval-augmented-generation","vector-database","vector-search","vectors"],"archived":false,"github_pushed_at":"2026-07-31T18:53:24+00:00","maintenance_label":"Active","stars_delta_30d":81,"url":"https://www.graphcanon.com/tools/startrail-org-leann","markdown_url":"https://www.graphcanon.com/tools/startrail-org-leann.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/startrail-org-leann","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=startrail-org-leann"}},{"type":"integrates_with","direction":"in","explanation":"Jan uses C/C++ inference for running LLM models and can potentially integrate with llama.cpp or other similar libraries.","successor_context":null,"tool":{"slug":"janhq-jan","name":"jan","tagline":"Jan is an open source alternative to ChatGPT that runs 100% offline on your computer.","github_url":"https://github.com/janhq/jan","owner":"janhq","repo":"jan","owner_avatar_url":"https://avatars.githubusercontent.com/u/102363196?v=4","primary_language":"TypeScript","stars":44518,"forks":3034,"topics":["chatgpt","gpt","llamacpp","llm","localai","open-source","self-hosted","tauri"],"archived":false,"github_pushed_at":"2026-09-18T08:22:25+00:00","maintenance_label":"Very active","stars_delta_30d":498,"url":"https://www.graphcanon.com/tools/janhq-jan","markdown_url":"https://www.graphcanon.com/tools/janhq-jan.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/janhq-jan","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=janhq-jan"}},{"type":"alternative","direction":"in","explanation":"Both PowerInfer and llama.cpp are focused on LLM inference, particularly for local deployment. While they serve similar purposes, they do so through different implementations and potentially with varying performance characteristics.","successor_context":null,"tool":{"slug":"tiiny-ai-powerinfer","name":"PowerInfer","tagline":"High-speed Large Language Model Serving for Local Deployment","github_url":"https://github.com/Tiiny-AI/PowerInfer","owner":"Tiiny-AI","repo":"PowerInfer","owner_avatar_url":"https://avatars.githubusercontent.com/u/256922953?v=4","primary_language":"C++","stars":9805,"forks":597,"topics":["large-language-models","llama","llm","llm-inference","local-inference"],"archived":false,"github_pushed_at":"2026-05-11T06:48:06+00:00","maintenance_label":"Slowing","stars_delta_30d":87,"url":"https://www.graphcanon.com/tools/tiiny-ai-powerinfer","markdown_url":"https://www.graphcanon.com/tools/tiiny-ai-powerinfer.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/tiiny-ai-powerinfer","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=tiiny-ai-powerinfer"}},{"type":"alternative","direction":"in","explanation":"OptiMate and llama.cpp both focus on optimizing LLMs, but they use different programming languages (Python vs C/C++).","successor_context":null,"tool":{"slug":"nebuly-ai-optimate","name":"optimate","tagline":"A collection of libraries to optimise AI model performances","github_url":"https://github.com/nebuly-ai/optimate","owner":"nebuly-ai","repo":"optimate","owner_avatar_url":"https://avatars.githubusercontent.com/u/83510798?v=4","primary_language":"Python","stars":8328,"forks":613,"topics":["ai","analytics","artificial-intelligence","deeplearning","large-language-models","llm"],"archived":false,"github_pushed_at":"2024-07-22T02:07:03+00:00","maintenance_label":"Dormant","stars_delta_30d":-1,"url":"https://www.graphcanon.com/tools/nebuly-ai-optimate","markdown_url":"https://www.graphcanon.com/tools/nebuly-ai-optimate.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/nebuly-ai-optimate","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=nebuly-ai-optimate"}},{"type":"alternative","direction":"in","explanation":"Both Video-LLaMA and llama.cpp offer inference capabilities for Large Language Models, but Video-LLaMA is geared towards instruction-tuned video understanding.","successor_context":null,"tool":{"slug":"damo-nlp-sg-video-llama","name":"Video-LLaMA","tagline":"Instruction-tuned Audio-Visual Language Model for Video Understanding","github_url":"https://github.com/DAMO-NLP-SG/Video-LLaMA","owner":"DAMO-NLP-SG","repo":"Video-LLaMA","owner_avatar_url":"https://avatars.githubusercontent.com/u/130957594?v=4","primary_language":"Python","stars":3141,"forks":287,"topics":["blip2","cross-modal-pretraining","large-language-models","llama","minigpt4","multi-modal-chatgpt","video-language-pretraining","vision-language-pretraining"],"archived":false,"github_pushed_at":"2024-06-04T07:06:41+00:00","maintenance_label":"Dormant","stars_delta_30d":2,"url":"https://www.graphcanon.com/tools/damo-nlp-sg-video-llama","markdown_url":"https://www.graphcanon.com/tools/damo-nlp-sg-video-llama.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/damo-nlp-sg-video-llama","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=damo-nlp-sg-video-llama"}},{"type":"depends_on","direction":"in","explanation":"Paddler uses a built-in llama.cpp engine for inference, thus Paddler depends on ggml-org-llama-cpp.","successor_context":null,"tool":{"slug":"intentee-paddler","name":"paddler","tagline":"Open-source LLM/VLM load balancer and serving platform for self-hosting at scale","github_url":"https://github.com/intentee/paddler","owner":"intentee","repo":"paddler","owner_avatar_url":"https://avatars.githubusercontent.com/u/215040511?v=4","primary_language":"Rust","stars":1674,"forks":98,"topics":["ai","llamacpp","llm","llmops","load-balancer"],"archived":false,"github_pushed_at":"2026-09-17T17:41:40+00:00","maintenance_label":"Very active","stars_delta_30d":11,"url":"https://www.graphcanon.com/tools/intentee-paddler","markdown_url":"https://www.graphcanon.com/tools/intentee-paddler.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/intentee-paddler","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=intentee-paddler"}},{"type":"related","direction":"in","explanation":"Both work on LLM inference but ggml focuses on low-level C/C++ implementation whereas petals provides a higher level distributed execution framework.","successor_context":null,"tool":{"slug":"bigscience-workshop-petals","name":"petals","tagline":"Distributed LLM Inference and Fine-Tuning","github_url":"https://github.com/bigscience-workshop/petals","owner":"bigscience-workshop","repo":"petals","owner_avatar_url":"https://avatars.githubusercontent.com/u/82455566?v=4","primary_language":"Python","stars":10579,"forks":650,"topics":["bloom","chatbot","deep-learning","distributed-systems","falcon","gpt","guanaco","language-models","large-language-models","llama","machine-learning","mixtral","neural-networks","nlp","pipeline-parallelism","pretrained-models","pytorch","tensor-parallelism","transformer","volunteer-computing"],"archived":false,"github_pushed_at":"2024-09-07T11:54:28+00:00","maintenance_label":"Dormant","stars_delta_30d":83,"url":"https://www.graphcanon.com/tools/bigscience-workshop-petals","markdown_url":"https://www.graphcanon.com/tools/bigscience-workshop-petals.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/bigscience-workshop-petals","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=bigscience-workshop-petals"}},{"type":"related","direction":"in","explanation":"Both Ollama and llama.cpp are related to LLM inference, but while Ollama seems to provide a broader ecosystem (models + infrastructure), llama.cpp is specific to C/C++ inference.","successor_context":null,"tool":{"slug":"ollama-ollama","name":"ollama","tagline":"Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.","github_url":"https://github.com/ollama/ollama","owner":"ollama","repo":"ollama","owner_avatar_url":"https://avatars.githubusercontent.com/u/151674099?v=4","primary_language":"Go","stars":181196,"forks":17922,"topics":["deepseek","gemma","gemma3","glm","go","golang","gpt-oss","llama","llama3","llm","llms","minimax","mistral","ollama","qwen"],"archived":false,"github_pushed_at":"2026-09-17T23:54:24+00:00","maintenance_label":"Very active","stars_delta_30d":3672,"url":"https://www.graphcanon.com/tools/ollama-ollama","markdown_url":"https://www.graphcanon.com/tools/ollama-ollama.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ollama-ollama","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ollama-ollama"}}],"neighbours":[{"slug":"ggml-org-whisper-cpp","name":"whisper.cpp","tagline":"C++ port of OpenAI's Whisper model for speech-to-text","github_url":"https://github.com/ggml-org/whisper.cpp","owner":"ggml-org","repo":"whisper.cpp","owner_avatar_url":"https://avatars.githubusercontent.com/u/134263123?v=4","primary_language":"C++","stars":53742,"forks":6172,"topics":["inference","openai","speech-recognition","speech-to-text","transformer","whisper"],"archived":false,"github_pushed_at":"2026-09-18T07:20:00+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/ggml-org-whisper-cpp","markdown_url":"https://www.graphcanon.com/tools/ggml-org-whisper-cpp.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ggml-org-whisper-cpp","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ggml-org-whisper-cpp","shared_categories":["inference-serving"]},{"slug":"mozilla-ai-llamafile","name":"llamafile","tagline":"Distribute and run LLMs with a single file.","github_url":"https://github.com/mozilla-ai/llamafile","owner":"mozilla-ai","repo":"llamafile","owner_avatar_url":"https://avatars.githubusercontent.com/u/129804596?v=4","primary_language":"C++","stars":25999,"forks":1598,"topics":["cross-platform","gguf","llama-cpp","local-ai","local-inference","local-llm","open-source-ai","single-file-executable","speech-to-text"],"archived":false,"github_pushed_at":"2026-09-16T09:14:28+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/mozilla-ai-llamafile","markdown_url":"https://www.graphcanon.com/tools/mozilla-ai-llamafile.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/mozilla-ai-llamafile","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=mozilla-ai-llamafile","shared_categories":["inference-serving"]},{"slug":"jundot-omlx","name":"omlx","tagline":"LLM inference server with continuous batching and SSD caching for Apple Silicon","github_url":"https://github.com/jundot/omlx","owner":"jundot","repo":"omlx","owner_avatar_url":"https://avatars.githubusercontent.com/u/64250138?v=4","primary_language":"Python","stars":21934,"forks":1899,"topics":["apple-silicon","inference-server","llm","macos","mlx","openai-api"],"archived":false,"github_pushed_at":"2026-09-20T02:23:26+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/jundot-omlx","markdown_url":"https://www.graphcanon.com/tools/jundot-omlx.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/jundot-omlx","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=jundot-omlx","shared_categories":["inference-serving"]},{"slug":"meta-llama-codellama","name":"codellama","tagline":"Inference code for CodeLlama models","github_url":"https://github.com/meta-llama/codellama","owner":"meta-llama","repo":"codellama","owner_avatar_url":"https://avatars.githubusercontent.com/u/153379578?v=4","primary_language":"Python","stars":16234,"forks":1930,"topics":[],"archived":true,"github_pushed_at":"2024-08-12T12:49:54+00:00","maintenance_label":"Archived","url":"https://www.graphcanon.com/tools/meta-llama-codellama","markdown_url":"https://www.graphcanon.com/tools/meta-llama-codellama.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/meta-llama-codellama","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=meta-llama-codellama","shared_categories":["inference-serving"]},{"slug":"nvidia-tensorrt-llm","name":"TensorRT-LLM","tagline":"Python API for defining and optimizing Large Language Models for efficient inference on NVIDIA GPUs","github_url":"https://github.com/NVIDIA/TensorRT-LLM","owner":"NVIDIA","repo":"TensorRT-LLM","owner_avatar_url":"https://avatars.githubusercontent.com/u/1728152?v=4","primary_language":"Python","stars":14645,"forks":2756,"topics":["blackwell","cuda","llm-serving","moe","pytorch"],"archived":false,"github_pushed_at":"2026-09-18T07:58:49+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/nvidia-tensorrt-llm","markdown_url":"https://www.graphcanon.com/tools/nvidia-tensorrt-llm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/nvidia-tensorrt-llm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=nvidia-tensorrt-llm","shared_categories":["inference-serving"]},{"slug":"0xplaygrounds-rig","name":"rig","tagline":"Build modular and scalable LLM Applications in Rust","github_url":"https://github.com/0xPlaygrounds/rig","owner":"0xPlaygrounds","repo":"rig","owner_avatar_url":"https://avatars.githubusercontent.com/u/93353392?v=4","primary_language":"Rust","stars":8679,"forks":974,"topics":["agent","ai","artificial-intelligence","automation","generative-ai","large-language-model","llm","llmops","rust","scalable-ai"],"archived":false,"github_pushed_at":"2026-09-19T23:48:13+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/0xplaygrounds-rig","markdown_url":"https://www.graphcanon.com/tools/0xplaygrounds-rig.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/0xplaygrounds-rig","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=0xplaygrounds-rig","shared_categories":[]},{"slug":"flashinfer-ai-flashinfer","name":"flashinfer","tagline":"FlashInfer is a kernel library for serving large language models","github_url":"https://github.com/flashinfer-ai/flashinfer","owner":"flashinfer-ai","repo":"flashinfer","owner_avatar_url":"https://avatars.githubusercontent.com/u/145061914?v=4","primary_language":"Cuda","stars":6452,"forks":1472,"topics":["attention","cuda","distributed-inference","gpu","jit","large-large-models","llm-inference","moe","nvidia","pytorch"],"archived":false,"github_pushed_at":"2026-09-19T01:31:07+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/flashinfer-ai-flashinfer","markdown_url":"https://www.graphcanon.com/tools/flashinfer-ai-flashinfer.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/flashinfer-ai-flashinfer","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=flashinfer-ai-flashinfer","shared_categories":["inference-serving"]},{"slug":"xlite-dev-awesome-llm-inference","name":"Awesome-LLM-Inference","tagline":"A curated list of LLM/VLM inference papers with codes","github_url":"https://github.com/xlite-dev/Awesome-LLM-Inference","owner":"xlite-dev","repo":"Awesome-LLM-Inference","owner_avatar_url":"https://avatars.githubusercontent.com/u/204302598?v=4","primary_language":"Python","stars":5508,"forks":435,"topics":["awesome-llm","deepseek","deepseek-r1","deepseek-v3","flash-attention","flash-attention-3","flash-mla","llm-inference","minimax-01","mla","paged-attention","qwen3","tensorrt-llm","vllm"],"archived":false,"github_pushed_at":"2026-08-14T12:23:49+00:00","maintenance_label":"Steady","url":"https://www.graphcanon.com/tools/xlite-dev-awesome-llm-inference","markdown_url":"https://www.graphcanon.com/tools/xlite-dev-awesome-llm-inference.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/xlite-dev-awesome-llm-inference","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=xlite-dev-awesome-llm-inference","shared_categories":["inference-serving"]},{"slug":"turboderp-exllama","name":"exllama","tagline":"Memory-efficient rewrite of HF transformers for Llama with quantized weights","github_url":"https://github.com/turboderp/exllama","owner":"turboderp","repo":"exllama","owner_avatar_url":"https://avatars.githubusercontent.com/u/11859846?v=4","primary_language":"Python","stars":2943,"forks":220,"topics":[],"archived":false,"github_pushed_at":"2023-09-30T19:06:04+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/turboderp-exllama","markdown_url":"https://www.graphcanon.com/tools/turboderp-exllama.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/turboderp-exllama","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=turboderp-exllama","shared_categories":["inference-serving"]},{"slug":"xllm-ai-xllm","name":"xllm","tagline":"A high-performance inference engine for LLM, VLM, DiT and REC models","github_url":"https://github.com/xLLM-AI/xllm","owner":"xLLM-AI","repo":"xllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/205719415?v=4","primary_language":"C++","stars":1577,"forks":300,"topics":["deepseek","glm","inference","inference-engine","large-language-models","llm-inference","qwen"],"archived":false,"github_pushed_at":"2026-09-19T13:15:59+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/xllm-ai-xllm","markdown_url":"https://www.graphcanon.com/tools/xllm-ai-xllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/xllm-ai-xllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=xllm-ai-xllm","shared_categories":["inference-serving"]},{"slug":"ngxson-wllama","name":"wllama","tagline":"WebAssembly binding for llama.cpp - Enabling on-browser LLM inference","github_url":"https://github.com/ngxson/wllama","owner":"ngxson","repo":"wllama","owner_avatar_url":"https://avatars.githubusercontent.com/u/7702203?v=4","primary_language":"TypeScript","stars":1196,"forks":121,"topics":["llama","llamacpp","llm","wasm","webassembly"],"archived":false,"github_pushed_at":"2026-09-06T12:17:08+00:00","maintenance_label":"Steady","url":"https://www.graphcanon.com/tools/ngxson-wllama","markdown_url":"https://www.graphcanon.com/tools/ngxson-wllama.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ngxson-wllama","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ngxson-wllama","shared_categories":["inference-serving"]},{"slug":"jmaczan-tiny-vllm","name":"tiny-vllm","tagline":"Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM","github_url":"https://github.com/jmaczan/tiny-vllm","owner":"jmaczan","repo":"tiny-vllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/18054202?v=4","primary_language":"C++","stars":1120,"forks":92,"topics":["ai","attention","batching","course","cpp","cuda","hpc","inference","llm","llm-inference","pagedattention","tiny-vllm","vllm"],"archived":false,"github_pushed_at":"2026-09-15T19:46:09+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/jmaczan-tiny-vllm","markdown_url":"https://www.graphcanon.com/tools/jmaczan-tiny-vllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/jmaczan-tiny-vllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=jmaczan-tiny-vllm","shared_categories":["inference-serving"]}]}}