{"data":{"slug":"zhihu-zhilight","name":"ZhiLight","tagline":"A highly optimized LLM inference acceleration engine for Llama and its variants.","github_url":"https://github.com/zhihu/ZhiLight","owner":"zhihu","repo":"ZhiLight","owner_avatar_url":"https://avatars.githubusercontent.com/u/409513?v=4","primary_language":"C++","stars":908,"forks":104,"topics":["cuda","deepseek-r1","gpt","inference-engine","llama","llm","llm-inference","llm-serving","model-serving","pytorch"],"archived":false,"github_pushed_at":"2026-03-18T02:47:55+00:00","maintenance_label":"Slowing","stars_delta_30d":3,"url":"https://www.graphcanon.com/tools/zhihu-zhilight","markdown_url":"https://www.graphcanon.com/tools/zhihu-zhilight.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/zhihu-zhilight","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=zhihu-zhilight","description":"A highly optimized LLM inference acceleration engine for Llama and its variants.","homepage_url":null,"license":"Apache-2.0","open_issues":6,"watchers":52,"ai_summary":"ZhiLight is an advanced tool designed to improve the efficiency of LLM inference tasks related to Llama models and their variations, using C++ as primary language with optimizations including CUDA integration. It aims to provide faster serving and inference capabilities for deep learning models.","readme_excerpt":"# Install directly\ncd ./ZhiLight && pip install -e .\n\n---\n\n## ✈️ Docker Image\nZhiLight only depends on the CUDA runtime, cuBLAS, NCCL, and a few Python packages in requirements.txt. You can use the image below for running or building it. You can also directly refer to docker/Dockerfile.\n```bash\ndocker pull ghcr.io/zhihu/zhilight/zhilight:0.4.8-cu124\n```","github_created_at":"2024-12-06T11:28:17+00:00","created_at":"2026-07-11T11:44:17.968332+00:00","updated_at":"2026-09-20T06:34:24.842629+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"cuda","name":"cuda"},{"slug":"deepseek-r1","name":"deepseek-r1"},{"slug":"gpt","name":"gpt"},{"slug":"inference-engine","name":"inference-engine"},{"slug":"llama","name":"llama"},{"slug":"llm","name":"llm"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"llm-serving","name":"llm-serving"}],"trust":{"provenance":{"is_fork":false,"github_id":899491463,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-19T18:32:08.004Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":185,"last_release_at":"2024-12-10T04:04:58Z","stars_delta_30d":3,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:44:19.042Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-19T18:32:09.341Z"},"languages":{"value":["c++"],"source":"github.language","observed_at":"2026-09-19T18:32:09.341Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-19T18:32:09.341Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium","summary":"The open-source version of ZhiLight is available under the Apache-2.0 license, allowing free use and modification."},"requirements":null,"constraints":{"pricing_model":"freemium"},"when_to_use":["Use ZhiLight if your application specifically requires optimization for Llama model variants, as it has specialized capabilities for this purpose.","Consider ZhiLight when you need to deploy on hardware that benefits from CUDA acceleration, given its deep integration with GPU technology."],"when_not_to_use":["Avoid using ZhiLight if your project relies on models other than Llama and its variants since the tool is optimized specifically for these models.","If your infrastructure does not include CUDA-compatible GPUs, or you prefer non-GPU-based acceleration solutions, then ZhiLight might not be advantageous."],"source":"enrich:decision_facts","observed_at":"2026-07-14T17:53:07.401Z"},"constraint_facets":{"pricing_model":"freemium"},"decision_summary":[{"label":"Pricing","value":"freemium - The open-source version of ZhiLight is available under the Apache-2.0 license, allowing free use and modification."},{"label":"Adopt for","value":"ZhiLight is an LLM inference acceleration engine aimed at enhancing serving and inference efficiency for Llama models using CUDA integration with C++ programming."}]}}