exllama
More memory-efficient rewrite of HF transformers for Llama with quantized weights
GraphCanon updated today · GitHub synced today
Trust & integrity
Full report- Maintenance
- Dormant (1014d since push)
- As of today · Source: github_public_v1
- Provenance
- Not a fork · Personal account
- As of today · Source: github_public_v1
- Security (OSV)
- 29 low (29 low)
- As of today · Source: osv@v1
Public GitHub metadata and optional OSV dependency scans. Signals, not a guarantee. Trust methodology.
Overview
ExLlama provides an optimized and more memory-efficient alternative to the HF transformers implementation specifically tailored for use with quantized Llama models, targeting hardware such as RTX series GPUs.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Jul 12, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Jul 12, 2026
- Languages
- python
Source: github.language · Jul 12, 2026
Categories
Tags
README
Hardware requirements
I am developing on an RTX 4090 and an RTX 3090-Ti. 30-series and later NVIDIA GPUs should be well supported, but anything Pascal or older with poor FP16 support isn't going to perform well. AutoGPTQ or GPTQ-for-LLaMa are better options at the moment for older GPUs. ROCm is also theoretically supported (via HIP) though I currently have no AMD devices to test or optimize on.
Docker
For security benefits and easier deployment, it is also possible to run the web UI in an isolated docker container. Note: the docker image currently only supports NVIDIA GPUs.
Requirements
It is recommended to run docker in rootless mode.