{"data":{"node":{"slug":"algorithmicsuperintelligence-optillm","name":"optillm","tagline":"Optimizing inference proxy for LLMs","github_url":"https://github.com/algorithmicsuperintelligence/optillm","owner":"algorithmicsuperintelligence","repo":"optillm","owner_avatar_url":"https://avatars.githubusercontent.com/u/238764598?v=4","primary_language":"Python","stars":4244,"forks":385,"topics":["agent","agentic-ai","agentic-framework","agentic-workflow","agents","api-gateway","chain-of-thought","genai","large-language-models","llm","llm-inference","llmapi","mixture-of-experts","moa","monte-carlo-tree-search","openai","openai-api","optimization","prompt-engineering","proxy-server"],"archived":false,"github_pushed_at":"2026-07-18T12:56:27+00:00","maintenance_label":"Steady","stars_delta_30d":67,"url":"https://www.graphcanon.com/tools/algorithmicsuperintelligence-optillm","markdown_url":"https://www.graphcanon.com/tools/algorithmicsuperintelligence-optillm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/algorithmicsuperintelligence-optillm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=algorithmicsuperintelligence-optillm"},"categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"agent","name":"agent"},{"slug":"agentic-ai","name":"agentic-ai"},{"slug":"genai","name":"genai"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"optimization","name":"optimization"},{"slug":"proxyservice","name":"proxyservice"}],"edges":[{"type":"related","direction":"out","explanation":"Both platforms allow users to run various LLMs, but they serve slightly different purposes; Ollama focuses on deployment setup while OptiLLM provides an optimizing proxy.","successor_context":null,"tool":{"slug":"ollama-ollama","name":"ollama","tagline":"Get up and running with various large language models using Ollama.","github_url":"https://github.com/ollama/ollama","owner":"ollama","repo":"ollama","owner_avatar_url":"https://avatars.githubusercontent.com/u/151674099?v=4","primary_language":"Go","stars":177524,"forks":17229,"topics":["deepseek","gemma","gemma3","glm","go","golang","gpt-oss","llama","llama3","llm","llms","minimax","mistral","ollama","qwen"],"archived":false,"github_pushed_at":"2026-07-31T23:59:29+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/ollama-ollama","markdown_url":"https://www.graphcanon.com/tools/ollama-ollama.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ollama-ollama","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ollama-ollama"}},{"type":"alternative","direction":"out","explanation":"OptiLLM and airllm both target improving the efficiency of large language model inference, but they serve slightly different niches; OptiLLM optimizes accuracy through various techniques without retraining, whereas airllm aims to enable lightweight GPU setups.","successor_context":null,"tool":{"slug":"lyogavin-airllm","name":"airllm","tagline":"AirLLM 70B inference with single 4GB GPU","github_url":"https://github.com/lyogavin/airllm","owner":"lyogavin","repo":"airllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/1113905?v=4","primary_language":"Jupyter Notebook","stars":24183,"forks":2722,"topics":["chinese-llm","chinese-nlp","finetune","generative-ai","instruct-gpt","instruction-set","llama","llm","lora","open-models","open-source","open-source-models","qlora"],"archived":false,"github_pushed_at":"2026-07-23T08:29:43+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/lyogavin-airllm","markdown_url":"https://www.graphcanon.com/tools/lyogavin-airllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/lyogavin-airllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=lyogavin-airllm"}},{"type":"integrates_with","direction":"out","explanation":"LitELM acts as an AI gateway supporting multiple models, and it could integrate with OptiLLM for optimized inference service across different LLMs.","successor_context":null,"tool":{"slug":"berriai-litellm","name":"litellm","tagline":"Python SDK and Proxy Server for calling multiple LLM APIs","github_url":"https://github.com/BerriAI/litellm","owner":"BerriAI","repo":"litellm","owner_avatar_url":"https://avatars.githubusercontent.com/u/121462774?v=4","primary_language":"Python","stars":55221,"forks":10231,"topics":["ai-gateway","anthropic","azure-openai","bedrock","gateway","langchain","litellm","llm","llm-gateway","llmops","mcp-gateway","openai","openai-proxy","rust","rust-ai","vertex-ai"],"archived":false,"github_pushed_at":"2026-08-01T05:53:28+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/berriai-litellm","markdown_url":"https://www.graphcanon.com/tools/berriai-litellm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/berriai-litellm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=berriai-litellm"}},{"type":"alternative","direction":"out","explanation":"OptiLLM and RTK both act as proxies to reduce token consumption and improve LLM efficiency, offering similar performance benefits.","successor_context":null,"tool":{"slug":"rtk-ai-rtk","name":"rtk","tagline":"CLI proxy reducing LLM token consumption by 60-90% on common dev commands","github_url":"https://github.com/rtk-ai/rtk","owner":"rtk-ai","repo":"rtk","owner_avatar_url":"https://avatars.githubusercontent.com/u/258253854?v=4","primary_language":"Rust","stars":76247,"forks":4795,"topics":["agentic-coding","ai-coding","anthropic","claude-code","cli","command-line-tool","cost-reduction","developer-tools","llm","open-source","productivity","rust","token-optimization"],"archived":false,"github_pushed_at":"2026-08-15T02:01:37+00:00","maintenance_label":"Very active","stars_delta_30d":4851,"url":"https://www.graphcanon.com/tools/rtk-ai-rtk","markdown_url":"https://www.graphcanon.com/tools/rtk-ai-rtk.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/rtk-ai-rtk","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=rtk-ai-rtk"}},{"type":"integrates_with","direction":"out","explanation":"OptiLLM could integrate with the sglang framework for serving large language models, providing both optimized performance and a robust serving infrastructure.","successor_context":null,"tool":{"slug":"sgl-project-sglang","name":"sglang","tagline":"High-performance serving framework for large language and multimodal models","github_url":"https://github.com/sgl-project/sglang","owner":"sgl-project","repo":"sglang","owner_avatar_url":"https://avatars.githubusercontent.com/u/147780389?v=4","primary_language":"Python","stars":31454,"forks":7720,"topics":["attention","blackwell","cuda","deepseek","diffusion","glm","gpt-oss","inference","llama","llm","minimax","moe","qwen","qwen-image","reinforcement-learning","transformer","vlm","wan"],"archived":false,"github_pushed_at":"2026-08-07T06:00:20+00:00","maintenance_label":"Very active","stars_delta_30d":1409,"url":"https://www.graphcanon.com/tools/sgl-project-sglang","markdown_url":"https://www.graphcanon.com/tools/sgl-project-sglang.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/sgl-project-sglang","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=sgl-project-sglang"}},{"type":"alternative","direction":"out","explanation":"Both OptiLLM and vLLM aim to optimize the performance of LLMs, but they approach it differently. While OptiLLM focuses on optimizing inference without requiring training or fine-tuning, vLLM provides an easy framework for serving LLMs efficiently.","successor_context":null,"tool":{"slug":"vllm-project-vllm","name":"vllm","tagline":"A high-throughput and memory-efficient inference and serving engine for LLMs","github_url":"https://github.com/vllm-project/vllm","owner":"vllm-project","repo":"vllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/136984999?v=4","primary_language":"Python","stars":87847,"forks":20135,"topics":["amd","blackwell","cuda","deepseek","deepseek-v3","gpt","gpt-oss","inference","kimi","llama","llm","llm-serving","model-serving","moe","openai","pytorch","qwen","qwen3","tpu","transformer"],"archived":false,"github_pushed_at":"2026-08-01T11:55:36+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/vllm-project-vllm","markdown_url":"https://www.graphcanon.com/tools/vllm-project-vllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/vllm-project-vllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=vllm-project-vllm"}},{"type":"related","direction":"out","explanation":null,"successor_context":null,"tool":{"slug":"vllm-project-vllm","name":"vllm","tagline":"A high-throughput and memory-efficient inference and serving engine for LLMs","github_url":"https://github.com/vllm-project/vllm","owner":"vllm-project","repo":"vllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/136984999?v=4","primary_language":"Python","stars":87847,"forks":20135,"topics":["amd","blackwell","cuda","deepseek","deepseek-v3","gpt","gpt-oss","inference","kimi","llama","llm","llm-serving","model-serving","moe","openai","pytorch","qwen","qwen3","tpu","transformer"],"archived":false,"github_pushed_at":"2026-08-01T11:55:36+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/vllm-project-vllm","markdown_url":"https://www.graphcanon.com/tools/vllm-project-vllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/vllm-project-vllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=vllm-project-vllm"}},{"type":"integrates_with","direction":"out","explanation":"OptiLLM could potentially integrate with litgpt, as it focuses on high-performance LLM deployment and pretraining, while OptiLLM can enhance inference performance by applying techniques that do not require retraining.","successor_context":null,"tool":{"slug":"lightning-ai-litgpt","name":"litgpt","tagline":"High-performance LLMs with recipes for pretraining, finetuning and deployment","github_url":"https://github.com/Lightning-AI/litgpt","owner":"Lightning-AI","repo":"litgpt","owner_avatar_url":"https://avatars.githubusercontent.com/u/58386951?v=4","primary_language":"Python","stars":13605,"forks":1483,"topics":["ai","artificial-intelligence","deep-learning","large-language-models","llm","llm-inference","llms"],"archived":false,"github_pushed_at":"2026-07-20T10:24:12+00:00","maintenance_label":"Active","stars_delta_30d":137,"url":"https://www.graphcanon.com/tools/lightning-ai-litgpt","markdown_url":"https://www.graphcanon.com/tools/lightning-ai-litgpt.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/lightning-ai-litgpt","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=lightning-ai-litgpt"}}],"neighbours":[{"slug":"vllm-project-vllm","name":"vllm","tagline":"A high-throughput and memory-efficient inference and serving engine for LLMs","github_url":"https://github.com/vllm-project/vllm","owner":"vllm-project","repo":"vllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/136984999?v=4","primary_language":"Python","stars":87847,"forks":20135,"topics":["amd","blackwell","cuda","deepseek","deepseek-v3","gpt","gpt-oss","inference","kimi","llama","llm","llm-serving","model-serving","moe","openai","pytorch","qwen","qwen3","tpu","transformer"],"archived":false,"github_pushed_at":"2026-08-01T11:55:36+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/vllm-project-vllm","markdown_url":"https://www.graphcanon.com/tools/vllm-project-vllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/vllm-project-vllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=vllm-project-vllm","shared_categories":["inference-serving"]},{"slug":"lyogavin-airllm","name":"airllm","tagline":"AirLLM 70B inference with single 4GB GPU","github_url":"https://github.com/lyogavin/airllm","owner":"lyogavin","repo":"airllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/1113905?v=4","primary_language":"Jupyter Notebook","stars":24183,"forks":2722,"topics":["chinese-llm","chinese-nlp","finetune","generative-ai","instruct-gpt","instruction-set","llama","llm","lora","open-models","open-source","open-source-models","qlora"],"archived":false,"github_pushed_at":"2026-07-23T08:29:43+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/lyogavin-airllm","markdown_url":"https://www.graphcanon.com/tools/lyogavin-airllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/lyogavin-airllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=lyogavin-airllm","shared_categories":["inference-serving"]},{"slug":"comet-ml-opik","name":"opik","tagline":"Debug, evaluate, and monitor your LLM applications with comprehensive tracing and production-ready dashboards","github_url":"https://github.com/comet-ml/opik","owner":"comet-ml","repo":"opik","owner_avatar_url":"https://avatars.githubusercontent.com/u/31487821?v=4","primary_language":"Python","stars":21177,"forks":1681,"topics":["evaluation","hacktoberfest","hacktoberfest2025","langchain","llama-index","llm","llm-evaluation","llm-observability","llmops","open-source","openai","playground","prompt-engineering"],"archived":false,"github_pushed_at":"2026-08-07T11:46:34+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/comet-ml-opik","markdown_url":"https://www.graphcanon.com/tools/comet-ml-opik.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/comet-ml-opik","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=comet-ml-opik","shared_categories":[]},{"slug":"jundot-omlx","name":"omlx","tagline":"LLM inference server with continuous batching and SSD caching for Apple Silicon","github_url":"https://github.com/jundot/omlx","owner":"jundot","repo":"omlx","owner_avatar_url":"https://avatars.githubusercontent.com/u/64250138?v=4","primary_language":"Python","stars":18679,"forks":1617,"topics":["apple-silicon","inference-server","llm","macos","mlx","openai-api"],"archived":false,"github_pushed_at":"2026-08-14T09:12:48+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/jundot-omlx","markdown_url":"https://www.graphcanon.com/tools/jundot-omlx.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/jundot-omlx","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=jundot-omlx","shared_categories":["inference-serving"]},{"slug":"nvidia-tensorrt-llm","name":"TensorRT-LLM","tagline":"Python API for defining and optimizing Large Language Models (LLMs) on NVIDIA GPUs","github_url":"https://github.com/NVIDIA/TensorRT-LLM","owner":"NVIDIA","repo":"TensorRT-LLM","owner_avatar_url":"https://avatars.githubusercontent.com/u/1728152?v=4","primary_language":"Python","stars":14317,"forks":2641,"topics":["blackwell","cuda","llm-serving","moe","pytorch"],"archived":false,"github_pushed_at":"2026-08-07T05:40:26+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/nvidia-tensorrt-llm","markdown_url":"https://www.graphcanon.com/tools/nvidia-tensorrt-llm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/nvidia-tensorrt-llm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=nvidia-tensorrt-llm","shared_categories":["inference-serving"]},{"slug":"lightning-ai-litgpt","name":"litgpt","tagline":"High-performance LLMs with recipes for pretraining, finetuning and deployment","github_url":"https://github.com/Lightning-AI/litgpt","owner":"Lightning-AI","repo":"litgpt","owner_avatar_url":"https://avatars.githubusercontent.com/u/58386951?v=4","primary_language":"Python","stars":13605,"forks":1483,"topics":["ai","artificial-intelligence","deep-learning","large-language-models","llm","llm-inference","llms"],"archived":false,"github_pushed_at":"2026-07-20T10:24:12+00:00","maintenance_label":"Steady","url":"https://www.graphcanon.com/tools/lightning-ai-litgpt","markdown_url":"https://www.graphcanon.com/tools/lightning-ai-litgpt.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/lightning-ai-litgpt","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=lightning-ai-litgpt","shared_categories":["inference-serving"]},{"slug":"luhengshiwo-llmforeverybody","name":"LLMForEverybody","tagline":"LLM knowledge sharing for everyone, essential reading before big model interviews","github_url":"https://github.com/luhengshiwo/LLMForEverybody","owner":"luhengshiwo","repo":"LLMForEverybody","owner_avatar_url":"https://avatars.githubusercontent.com/u/13251733?v=4","primary_language":"Jupyter Notebook","stars":7167,"forks":662,"topics":["agent","interview-practice","interview-questions","learnllm","llm","rag"],"archived":false,"github_pushed_at":"2026-08-17T02:41:01+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/luhengshiwo-llmforeverybody","markdown_url":"https://www.graphcanon.com/tools/luhengshiwo-llmforeverybody.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/luhengshiwo-llmforeverybody","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=luhengshiwo-llmforeverybody","shared_categories":[]},{"slug":"flashinfer-ai-flashinfer","name":"flashinfer","tagline":"FlashInfer is a kernel library for serving large language models","github_url":"https://github.com/flashinfer-ai/flashinfer","owner":"flashinfer-ai","repo":"flashinfer","owner_avatar_url":"https://avatars.githubusercontent.com/u/145061914?v=4","primary_language":"Python","stars":6024,"forks":1196,"topics":["attention","cuda","distributed-inference","gpu","jit","large-large-models","llm-inference","moe","nvidia","pytorch"],"archived":false,"github_pushed_at":"2026-07-25T05:01:59+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/flashinfer-ai-flashinfer","markdown_url":"https://www.graphcanon.com/tools/flashinfer-ai-flashinfer.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/flashinfer-ai-flashinfer","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=flashinfer-ai-flashinfer","shared_categories":["inference-serving"]},{"slug":"xlite-dev-awesome-llm-inference","name":"Awesome-LLM-Inference","tagline":"A curated list of LLM/VLM inference papers with codes","github_url":"https://github.com/xlite-dev/Awesome-LLM-Inference","owner":"xlite-dev","repo":"Awesome-LLM-Inference","owner_avatar_url":"https://avatars.githubusercontent.com/u/204302598?v=4","primary_language":"Python","stars":5415,"forks":428,"topics":["awesome-llm","deepseek","deepseek-r1","deepseek-v3","flash-attention","flash-attention-3","flash-mla","llm-inference","minimax-01","mla","paged-attention","qwen3","tensorrt-llm","vllm"],"archived":false,"github_pushed_at":"2026-06-23T03:48:43+00:00","maintenance_label":"Steady","url":"https://www.graphcanon.com/tools/xlite-dev-awesome-llm-inference","markdown_url":"https://www.graphcanon.com/tools/xlite-dev-awesome-llm-inference.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/xlite-dev-awesome-llm-inference","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=xlite-dev-awesome-llm-inference","shared_categories":["inference-serving"]},{"slug":"eth-sri-lmql","name":"lmql","tagline":"A language for constraint-guided and efficient LLM programming.","github_url":"https://github.com/eth-sri/lmql","owner":"eth-sri","repo":"lmql","owner_avatar_url":"https://avatars.githubusercontent.com/u/5363413?v=4","primary_language":"Python","stars":4203,"forks":221,"topics":["chatgpt","huggingface","language-model","programming-language"],"archived":false,"github_pushed_at":"2025-05-22T07:32:31+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/eth-sri-lmql","markdown_url":"https://www.graphcanon.com/tools/eth-sri-lmql.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/eth-sri-lmql","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=eth-sri-lmql","shared_categories":[]},{"slug":"turboderp-exllama","name":"exllama","tagline":"Memory-efficient rewrite of HF transformers for Llama with quantized weights","github_url":"https://github.com/turboderp/exllama","owner":"turboderp","repo":"exllama","owner_avatar_url":"https://avatars.githubusercontent.com/u/11859846?v=4","primary_language":"Python","stars":2937,"forks":220,"topics":[],"archived":false,"github_pushed_at":"2023-09-30T19:06:04+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/turboderp-exllama","markdown_url":"https://www.graphcanon.com/tools/turboderp-exllama.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/turboderp-exllama","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=turboderp-exllama","shared_categories":["inference-serving"]},{"slug":"huangowen-awesome-llm-compression","name":"Awesome-LLM-Compression","tagline":"Awesome LLM compression research papers and tools to accelerate LLM training and inference.","github_url":"https://github.com/HuangOwen/Awesome-LLM-Compression","owner":"HuangOwen","repo":"Awesome-LLM-Compression","owner_avatar_url":"https://avatars.githubusercontent.com/u/24937399?v=4","primary_language":null,"stars":1859,"forks":129,"topics":[],"archived":false,"github_pushed_at":"2026-06-30T15:26:46+00:00","maintenance_label":"Steady","url":"https://www.graphcanon.com/tools/huangowen-awesome-llm-compression","markdown_url":"https://www.graphcanon.com/tools/huangowen-awesome-llm-compression.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/huangowen-awesome-llm-compression","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=huangowen-awesome-llm-compression","shared_categories":["inference-serving"]}]}}