{"data":{"slug":"turboderp-exllama","name":"exllama","tagline":"Memory-efficient rewrite of HF transformers for Llama with quantized weights","github_url":"https://github.com/turboderp/exllama","owner":"turboderp","repo":"exllama","owner_avatar_url":"https://avatars.githubusercontent.com/u/11859846?v=4","primary_language":"Python","stars":2943,"forks":220,"topics":[],"archived":false,"github_pushed_at":"2023-09-30T19:06:04+00:00","maintenance_label":"Dormant","stars_delta_30d":6,"url":"https://www.graphcanon.com/tools/turboderp-exllama","markdown_url":"https://www.graphcanon.com/tools/turboderp-exllama.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/turboderp-exllama","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=turboderp-exllama","description":"A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.","homepage_url":null,"license":"MIT","open_issues":65,"watchers":34,"ai_summary":"turboderp/exllama provides an optimized version of the Hugging Face Transformers implementation specifically for the Llama model, offering improved memory usage and support for quantized weights. The software is designed to work efficiently on NVIDIA GPUs from the 30-series onwards.","readme_excerpt":"## Hardware requirements\n\nI am developing on an RTX 4090 and an RTX 3090-Ti. 30-series and later NVIDIA GPUs should be well supported, but\nanything Pascal or older with poor FP16 support isn't going to perform well. \n[AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) or [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa)\nare better options at the moment for older GPUs. ROCm is also theoretically supported (via HIP) though I currently \nhave no AMD devices to test or optimize on.\n\n---\n\n## Docker\n\nFor security benefits and easier deployment, it is also possible to run the web UI in an isolated docker container. Note: the docker image currently only supports NVIDIA GPUs.\n\n---\n\n### Requirements\n\n- [Docker](https://docs.docker.com/engine/install/)\n- [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)\n\nIt is recommended to run docker in [rootless mode](https://docs.docker.com/engine/security/rootless/).","github_created_at":"2023-05-04T23:10:58+00:00","created_at":"2026-07-11T10:37:10.048724+00:00","updated_at":"2026-09-20T04:23:26.688194+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"docker","name":"docker"},{"slug":"llama-model","name":"llama model"},{"slug":"memory-efficient","name":"memory-efficient"},{"slug":"nvidia-gpu","name":"nvidia gpu"},{"slug":"python","name":"python"},{"slug":"quantized-weights","name":"quantized weights"}],"trust":{"provenance":{"is_fork":false,"github_id":636467087,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-07T00:01:13.522Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1072,"last_release_at":null,"stars_delta_30d":6,"open_issues_delta_30d":0},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":29,"high_count":0,"last_scan_at":"2026-07-11T10:37:11.794Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-07T00:01:14.000Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-09-07T00:01:14.000Z","managed_saas":false},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-09-07T00:01:14.000Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-09-07T00:01:14.000Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-09-07T00:01:14.000Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["- When deploying LLaMa models on NVIDIA GPUs from the 30-series or later that have strong FP16 support.","- For environments where memory efficiency is critical and you utilize quantized weights for your LLaMa model."],"when_not_to_use":["- If you are operating older GPUs such as Pascal series, which lack robust FP16 support; alternatives like AutoGPTQ might perform better.","- In scenarios that involve AMD GPU hardware (due to limited testing and optimization efforts)."],"source":"enrich:decision_facts","observed_at":"2026-07-12T13:32:50.892Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"ExLlama provides a memory-efficient implementation of the LLaMa model with support for quantized weights, primarily aimed at users with NVIDIA GPUs from the 30-series onwards."}]}}