{"data":{"slug":"turboderp-exllama","name":"exllama","tagline":"More memory-efficient rewrite of HF transformers for Llama with quantized weights","github_url":"https://github.com/turboderp/exllama","owner":"turboderp","repo":"exllama","owner_avatar_url":"https://avatars.githubusercontent.com/u/11859846?v=4","primary_language":"Python","stars":2930,"forks":223,"topics":[],"archived":false,"github_pushed_at":"2023-09-30T19:06:04+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/turboderp-exllama","markdown_url":"https://www.graphcanon.com/tools/turboderp-exllama.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/turboderp-exllama","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=turboderp-exllama","description":"A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.","homepage_url":null,"license":"MIT","open_issues":65,"watchers":34,"ai_summary":"ExLlama provides an optimized and more memory-efficient alternative to the HF transformers implementation specifically tailored for use with quantized Llama models, targeting hardware such as RTX series GPUs.","readme_excerpt":"## Hardware requirements\n\nI am developing on an RTX 4090 and an RTX 3090-Ti. 30-series and later NVIDIA GPUs should be well supported, but\nanything Pascal or older with poor FP16 support isn't going to perform well. \n[AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) or [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa)\nare better options at the moment for older GPUs. ROCm is also theoretically supported (via HIP) though I currently \nhave no AMD devices to test or optimize on.\n\n---\n\n## Docker\n\nFor security benefits and easier deployment, it is also possible to run the web UI in an isolated docker container. Note: the docker image currently only supports NVIDIA GPUs.\n\n---\n\n### Requirements\n\n- [Docker](https://docs.docker.com/engine/install/)\n- [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)\n\nIt is recommended to run docker in [rootless mode](https://docs.docker.com/engine/security/rootless/).","github_created_at":"2023-05-04T23:10:58+00:00","created_at":"2026-07-11T10:37:10.048724+00:00","updated_at":"2026-07-12T02:39:33.671476+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"docker-container-support","name":"docker container support"},{"slug":"gpu-optimization","name":"gpu optimization"},{"slug":"memory-efficiency","name":"memory efficiency"},{"slug":"nvidia-support","name":"nvidia support"},{"slug":"quantization","name":"quantization"}],"trust":{"provenance":{"is_fork":false,"github_id":636467087,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:37:10.724Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1014,"last_release_at":null},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":29,"high_count":0,"last_scan_at":"2026-07-11T10:37:11.794Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-12T02:39:33.601Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-07-12T02:39:33.601Z","managed_saas":false},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-12T02:39:33.601Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-07-12T02:39:33.601Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-12T02:39:33.601Z"}}}}