airllm

lyogavin/airllm

Repository for running large language models with reduced memory usage on limited GPU hardware.

GitHub

22k

Stars

2.6k

Forks

106

Open issues

227

Watchers

Jupyter Notebook Apache-2.0Last pushed Jul 7, 2026

Overview

AirLLM allows efficient inference of large language models like 70B parameter sizes using only a single 4GB GPU, without applying techniques such as quantization, distillation, or pruning. It supports various large-scale models and enhances performance capabilities through continuous updates focusing on model optimizations.

Relationships

Alternatives

vllm

Integrates with

transformers

Chinese-LLaMA-Alpaca

Similar tools

open-webui

open-webui/open-webui

User-friendly AI Interface (Supports Ollama, OpenAI API, ...)

145k 21kPython

LLMs-from-scratch

rasbt/LLMs-from-scratch

Implement a ChatGPT-like LLM in PyTorch from scratch

99k 15kJupyter Notebook

rtk

rtk-ai/rtk

CLI proxy reducing LLM token consumption by 60-90%

69k 4.3kRust

unsloth

unslothai/unsloth

Unsloth Studio is a web UI for training and running open models locally.

68k 6.1kPython

Install

git clone https://github.com/lyogavin/airllm

See alternatives to airllm

README

Quickstart | Configurations | MacOS | Example notebooks | FAQ

AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run 405B Llama 3.1 on 8GB, and DeepSeek-V3 (671B) on ~12GB.

AI Agents Recommendation:

Updates

[2026/06] v3.0: FP8 model support + the latest models. Run DeepSeek-V3 (671B) on ~12GB and Qwen3-235B on ~3GB, plus Qwen3, Llama 3.x/4, DeepSeek V2/V3, Phi-4, Gemma and more — all through a single AutoModel.

[2024/08/20] v2.11.0: Support Qwen2.5

[2024/08/18] v2.10.1 Support CPU inference. Support non sharded models. Thanks @NavodPeiris for the great work!

[2024/07/30] Support Llama3.1 405B (example notebook). Support 8bit/4bit quantization.

[2024/04/20] AirLLM supports Llama3 natively already. Run Llama3 70B on 4GB single GPU.

[2023/12/25] v2.8.2: Support MacOS running 70B large language models.

[2023/12/20] v2.7: Support AirLLMMixtral.

[2023/12/20] v2.6: Added AutoModel, automatically detect model type, no need to provide model class to initialize model.

[2023/12/18] v2.5: added prefetching to overlap the model loading and compute. 10% speed improvement.

[2023/12/03] added support of ChatGLM, QWen, Baichuan, Mistral, InternLM!

[2023/12/02] added support for safetensors. Now support all top 10 models in open llm leaderboard.

[2023/12/01] airllm 2.0. Support compressions: 3x run time speed up!

[2023/11/20] airllm Initial version!

Quickstart

1. Install package

First, install the airllm pip package.

pip install airllm

2. Inference

Then, initialize AirLLMLlama2, pass in the huggingface repo ID of the model being used, or the local path, and inference can be performed similar to a regular transformer model.

(You can also specify the path to save the splitted layered model through layer_shards_saving_path when init AirLLMLlama2.

from airllm import AutoModel

MAX_LENGTH = 128
# just pass a hugging face repo id — works with almost any popular model:
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

# go bigger with the exact same one line:
#model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B")     # 235B, runs in ~3GB
#model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3")  # 671B, runs in ~12GB

# or use a model's local path...
#model = AutoModel.from_pretrained("/home/ubuntu/.cache/huggingface/hub/models--Qwen--Qwen3-32B/snapshots/...")

input_text = [
        'What is the capital of United States?',
        #'I like',
    ]

input_tokens = model.tokenizer(input_text,
    return_tensors="pt", 
    return_attention_mask=False, 
    truncation=True, 
    max_length=MAX_LENGTH, 
    padding=False)
           
generation_output = model.generate(
    input