exllama
Memory-efficient rewrite of HF transformers for Llama with quantized weights
GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026
90views this month
Decision brief
ExLlama provides a memory-efficient implementation of the LLaMa model with support for quantized weights, primarily aimed at users with NVIDIA GPUs from the 30-series onwards.
Good fit when
- - When deploying LLaMa models on NVIDIA GPUs from the 30-series or later that have strong FP16 support.
- - For environments where memory efficiency is critical and you utilize quantized weights for your LLaMa model.
Avoid when
- - If you are operating older GPUs such as Pascal series, which lack robust FP16 support; alternatives like AutoGPTQ might perform better.
- - In scenarios that involve AMD GPU hardware (due to limited testing and optimization efforts).
Observed Jul 12, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (1072d since push)
- As of Sep 7, 2026
- Provenance
- Not a fork · Personal account
- As of Sep 7, 2026
- Security (OSV)
- 29 low (29 low)
- As of Jul 11, 2026
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install exllamaSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
turboderp/exllama provides an optimized version of the Hugging Face Transformers implementation specifically for the Llama model, offering improved memory usage and support for quantized weights. The software is designed to work efficiently on NVIDIA GPUs from the 30-series onwards.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Sep 7, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Sep 7, 2026
- Languages
- python
Source: github.language · Sep 7, 2026
Categories
Tags
README
Hardware requirements I am developing on an RTX 4090 and an RTX 3090 Ti. 30 series and later NVIDIA GPUs should be well supported, but anything Pascal or older with poor FP16 support isn't going to perform well. AutoGPTQ or GPTQ for LLaMa are better options at the moment for olde...
For agents
This page has a .md twin and JSON over the API.