exllama
Memory-efficient rewrite of HF transformers for Llama with quantized weights
GraphCanon updated 1w · GitHub synced 1w
Decision brief
ExLlama provides a memory-efficient implementation of the LLaMa model with support for quantized weights, primarily aimed at users with NVIDIA GPUs from the 30-series onwards.
Good fit when
- - When deploying LLaMa models on NVIDIA GPUs from the 30-series or later that have strong FP16 support.
- - For environments where memory efficiency is critical and you utilize quantized weights for your LLaMa model.
Avoid when
- - If you are operating older GPUs such as Pascal series, which lack robust FP16 support; alternatives like AutoGPTQ might perform better.
- - In scenarios that involve AMD GPU hardware (due to limited testing and optimization efforts).
Observed Jul 12, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (1041d since push)
- As of 1w
- Provenance
- Not a fork · Personal account
- As of 1w
- Security (OSV)
- 29 low (29 low)
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install exllamaSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
turboderp/exllama provides an optimized version of the Hugging Face Transformers implementation specifically for the Llama model, offering improved memory usage and support for quantized weights. The software is designed to work efficiently on NVIDIA GPUs from the 30-series onwards.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Aug 7, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Aug 7, 2026
- Languages
- python
Source: github.language · Aug 7, 2026
Categories
Tags
README
Hardware requirements
I am developing on an RTX 4090 and an RTX 3090-Ti. 30-series and later NVIDIA GPUs should be well supported, but anything Pascal or older with poor FP16 support isn't going to perform well. AutoGPTQ or GPTQ-for-LLaMa are better options at the moment for older GPUs. ROCm is also theoretically supported (via HIP) though I currently have no AMD devices to test or optimize on.
Docker
For security benefits and easier deployment, it is also possible to run the web UI in an isolated docker container. Note: the docker image currently only supports NVIDIA GPUs.
Requirements
It is recommended to run docker in rootless mode.
For agents
This page has a .md twin and JSON over the API.