{"data":{"node":{"slug":"tiiny-ai-powerinfer","name":"PowerInfer","tagline":"High-speed Large Language Model Serving for Local Deployment","github_url":"https://github.com/Tiiny-AI/PowerInfer","owner":"Tiiny-AI","repo":"PowerInfer","owner_avatar_url":"https://avatars.githubusercontent.com/u/256922953?v=4","primary_language":"C++","stars":9805,"forks":597,"topics":["large-language-models","llama","llm","llm-inference","local-inference"],"archived":false,"github_pushed_at":"2026-05-11T06:48:06+00:00","maintenance_label":"Slowing","stars_delta_30d":87,"url":"https://www.graphcanon.com/tools/tiiny-ai-powerinfer","markdown_url":"https://www.graphcanon.com/tools/tiiny-ai-powerinfer.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/tiiny-ai-powerinfer","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=tiiny-ai-powerinfer"},"categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"large-language-models","name":"large-language-models"},{"slug":"llama","name":"llama"},{"slug":"llm","name":"llm"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"local-inference","name":"local-inference"}],"edges":[{"type":"integrates_with","direction":"out","explanation":"PowerInfer could be used as an inference backend for ollama to provide efficient local LLM running capabilities.","successor_context":null,"tool":{"slug":"ollama-ollama","name":"ollama","tagline":"Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.","github_url":"https://github.com/ollama/ollama","owner":"ollama","repo":"ollama","owner_avatar_url":"https://avatars.githubusercontent.com/u/151674099?v=4","primary_language":"Go","stars":181196,"forks":17922,"topics":["deepseek","gemma","gemma3","glm","go","golang","gpt-oss","llama","llama3","llm","llms","minimax","mistral","ollama","qwen"],"archived":false,"github_pushed_at":"2026-09-17T23:54:24+00:00","maintenance_label":"Very active","stars_delta_30d":3672,"url":"https://www.graphcanon.com/tools/ollama-ollama","markdown_url":"https://www.graphcanon.com/tools/ollama-ollama.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ollama-ollama","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ollama-ollama"}},{"type":"integrates_with","direction":"out","explanation":"PowerInfer, as a high-speed LLM serving solution, can potentially integrate with sglang's broader framework for both large language and multimodal model serving.","successor_context":null,"tool":{"slug":"sgl-project-sglang","name":"sglang","tagline":"SGLang is a high-performance serving framework for large language models and multimodal models.","github_url":"https://github.com/sgl-project/sglang","owner":"sgl-project","repo":"sglang","owner_avatar_url":"https://avatars.githubusercontent.com/u/147780389?v=4","primary_language":"Python","stars":36123,"forks":8969,"topics":["attention","blackwell","cuda","deepseek","diffusion","glm","gpt-oss","inference","llama","llm","minimax","moe","qwen","qwen-image","reinforcement-learning","transformer","vlm","wan"],"archived":false,"github_pushed_at":"2026-09-18T08:40:33+00:00","maintenance_label":"Very active","stars_delta_30d":4669,"url":"https://www.graphcanon.com/tools/sgl-project-sglang","markdown_url":"https://www.graphcanon.com/tools/sgl-project-sglang.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/sgl-project-sglang","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=sgl-project-sglang"}},{"type":"integrates_with","direction":"out","explanation":"PowerInfer could potentially integrate with MLC LLM as it provides an engine for deploying language models across various devices and contexts, which complements PowerInfer's deployment capabilities.","successor_context":null,"tool":{"slug":"mlc-ai-mlc-llm","name":"mlc-llm","tagline":"Universal LLM Deployment Engine with ML Compilation","github_url":"https://github.com/mlc-ai/mlc-llm","owner":"mlc-ai","repo":"mlc-llm","owner_avatar_url":"https://avatars.githubusercontent.com/u/106173866?v=4","primary_language":"Python","stars":23063,"forks":2111,"topics":["language-model","llm","machine-learning-compilation","tvm"],"archived":false,"github_pushed_at":"2026-07-31T03:03:18+00:00","maintenance_label":"Active","stars_delta_30d":103,"url":"https://www.graphcanon.com/tools/mlc-ai-mlc-llm","markdown_url":"https://www.graphcanon.com/tools/mlc-ai-mlc-llm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/mlc-ai-mlc-llm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=mlc-ai-mlc-llm"}},{"type":"alternative","direction":"out","explanation":"PowerInfer is an alternative to vLLM because both projects aim to serve LLMs efficiently, especially for local deployments where speed and performance are key.","successor_context":null,"tool":{"slug":"vllm-project-vllm","name":"vllm","tagline":"A high-throughput and memory-efficient inference and serving engine for LLMs","github_url":"https://github.com/vllm-project/vllm","owner":"vllm-project","repo":"vllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/136984999?v=4","primary_language":"Python","stars":92075,"forks":22363,"topics":["amd","blackwell","cuda","deepseek","deepseek-v3","gpt","gpt-oss","inference","kimi","llama","llm","llm-serving","model-serving","moe","openai","pytorch","qwen","qwen3","tpu","transformer"],"archived":false,"github_pushed_at":"2026-09-18T07:28:54+00:00","maintenance_label":"Very active","stars_delta_30d":4228,"url":"https://www.graphcanon.com/tools/vllm-project-vllm","markdown_url":"https://www.graphcanon.com/tools/vllm-project-vllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/vllm-project-vllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=vllm-project-vllm"}},{"type":"alternative","direction":"out","explanation":"Both PowerInfer and llama.cpp are focused on LLM inference, particularly for local deployment. While they serve similar purposes, they do so through different implementations and potentially with varying performance characteristics.","successor_context":null,"tool":{"slug":"ggml-org-llama-cpp","name":"llama.cpp","tagline":"LLM inference in C/C++","github_url":"https://github.com/ggml-org/llama.cpp","owner":"ggml-org","repo":"llama.cpp","owner_avatar_url":"https://avatars.githubusercontent.com/u/134263123?v=4","primary_language":"C++","stars":128639,"forks":23359,"topics":["ggml"],"archived":false,"github_pushed_at":"2026-09-18T08:13:37+00:00","maintenance_label":"Very active","stars_delta_30d":5698,"url":"https://www.graphcanon.com/tools/ggml-org-llama-cpp","markdown_url":"https://www.graphcanon.com/tools/ggml-org-llama-cpp.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ggml-org-llama-cpp","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ggml-org-llama-cpp"}}],"neighbours":[{"slug":"ggml-org-llama-cpp","name":"llama.cpp","tagline":"LLM inference in C/C++","github_url":"https://github.com/ggml-org/llama.cpp","owner":"ggml-org","repo":"llama.cpp","owner_avatar_url":"https://avatars.githubusercontent.com/u/134263123?v=4","primary_language":"C++","stars":128639,"forks":23359,"topics":["ggml"],"archived":false,"github_pushed_at":"2026-09-18T08:13:37+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/ggml-org-llama-cpp","markdown_url":"https://www.graphcanon.com/tools/ggml-org-llama-cpp.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ggml-org-llama-cpp","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ggml-org-llama-cpp","shared_categories":["inference-serving"]},{"slug":"xorbitsai-inference","name":"inference","tagline":"Unified production-ready inference API for various LLMs and models","github_url":"https://github.com/xorbitsai/inference","owner":"xorbitsai","repo":"inference","owner_avatar_url":"https://avatars.githubusercontent.com/u/109655068?v=4","primary_language":"Python","stars":9575,"forks":866,"topics":["artificial-intelligence","deployment","diffusers","gemma","glm","glm-5-3","inference","kimi","kimi-k3","llama-cpp","llamacpp","llm","machine-learning","openai-api","pytorch","qwen","sglang","transformers","vllm","whisper"],"archived":false,"github_pushed_at":"2026-09-18T06:04:24+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/xorbitsai-inference","markdown_url":"https://www.graphcanon.com/tools/xorbitsai-inference.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/xorbitsai-inference","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=xorbitsai-inference","shared_categories":["inference-serving"]},{"slug":"flashinfer-ai-flashinfer","name":"flashinfer","tagline":"FlashInfer is a kernel library for serving large language models","github_url":"https://github.com/flashinfer-ai/flashinfer","owner":"flashinfer-ai","repo":"flashinfer","owner_avatar_url":"https://avatars.githubusercontent.com/u/145061914?v=4","primary_language":"Cuda","stars":6452,"forks":1472,"topics":["attention","cuda","distributed-inference","gpu","jit","large-large-models","llm-inference","moe","nvidia","pytorch"],"archived":false,"github_pushed_at":"2026-09-19T01:31:07+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/flashinfer-ai-flashinfer","markdown_url":"https://www.graphcanon.com/tools/flashinfer-ai-flashinfer.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/flashinfer-ai-flashinfer","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=flashinfer-ai-flashinfer","shared_categories":["inference-serving"]},{"slug":"ghimiresunil-llm-powerhouse-a-curated-guide-for-large-language-models-with-custom-training-and-inferencing","name":"LLM-PowerHouse-A-Curated-Guide-for-Large-Language-Models-with-Custom-Training-and-Inferencing","tagline":"Curated tutorials and best practices for LLM custom training and inferencing","github_url":"https://github.com/ghimiresunil/LLM-PowerHouse-A-Curated-Guide-for-Large-Language-Models-with-Custom-Training-and-Inferencing","owner":"ghimiresunil","repo":"LLM-PowerHouse-A-Curated-Guide-for-Large-Language-Models-with-Custom-Training-and-Inferencing","owner_avatar_url":"https://avatars.githubusercontent.com/u/40186859?v=4","primary_language":"Jupyter Notebook","stars":733,"forks":121,"topics":["bert","huggingface","large-language-models","llm-inference","llm-training","llm-tutorials","open-source","open-source-llm","transformers"],"archived":false,"github_pushed_at":"2026-03-13T12:14:27+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/ghimiresunil-llm-powerhouse-a-curated-guide-for-large-language-models-with-custom-training-and-inferencing","markdown_url":"https://www.graphcanon.com/tools/ghimiresunil-llm-powerhouse-a-curated-guide-for-large-language-models-with-custom-training-and-inferencing.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ghimiresunil-llm-powerhouse-a-curated-guide-for-large-language-models-with-custom-training-and-inferencing","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ghimiresunil-llm-powerhouse-a-curated-guide-for-large-language-models-with-custom-training-and-inferencing","shared_categories":["inference-serving"]},{"slug":"underneathall-pinferencia","name":"pinferencia","tagline":"Python library for simplest model inference server","github_url":"https://github.com/underneathall/pinferencia","owner":"underneathall","repo":"pinferencia","owner_avatar_url":"https://avatars.githubusercontent.com/u/76835515?v=4","primary_language":"Python","stars":543,"forks":83,"topics":["ai","artificial-intelligence","computer-vision","data-science","deep-learning","huggingface","inference","inference-server","machine-learning","model-deployment","model-serving","modelserver","nlp","paddlepaddle","predict","python","pytorch","serving","tensorflow","transformers"],"archived":false,"github_pushed_at":"2023-02-14T22:50:48+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/underneathall-pinferencia","markdown_url":"https://www.graphcanon.com/tools/underneathall-pinferencia.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/underneathall-pinferencia","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=underneathall-pinferencia","shared_categories":["inference-serving"]},{"slug":"hpcaitech-swiftinfer","name":"SwiftInfer","tagline":"Efficient AI Inference Serving","github_url":"https://github.com/hpcaitech/SwiftInfer","owner":"hpcaitech","repo":"SwiftInfer","owner_avatar_url":"https://avatars.githubusercontent.com/u/88699314?v=4","primary_language":"Python","stars":474,"forks":31,"topics":["artificial-intelligence","deep-learning","gpt","inference","llama","llama2","llm-inference","llm-serving"],"archived":false,"github_pushed_at":"2024-01-08T09:18:42+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/hpcaitech-swiftinfer","markdown_url":"https://www.graphcanon.com/tools/hpcaitech-swiftinfer.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/hpcaitech-swiftinfer","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=hpcaitech-swiftinfer","shared_categories":["inference-serving"]}]}}