---
title: "airllm"
type: "tool"
slug: "lyogavin-airllm"
canonical_url: "https://www.graphcanon.com/tools/lyogavin-airllm"
github_url: "https://github.com/lyogavin/airllm"
homepage_url: null
stars: 22229
forks: 2556
primary_language: "Jupyter Notebook"
license: "Apache-2.0"
categories: ["inference-serving", "model-training"]
tags: ["llama", "chinese-llm", "llm", "instruct-gpt", "generative-ai", "instruction-set", "finetune", "chinese-nlp"]
updated_at: "2026-07-07T19:43:04.411452+00:00"
---

# airllm

> Repository for running large language models with reduced memory usage on limited GPU hardware.

AirLLM allows efficient inference of large language models like 70B parameter sizes using only a single 4GB GPU, without applying techniques such as quantization, distillation, or pruning. It supports various large-scale models and enhances performance capabilities through continuous updates focusing on model optimizations.

## Facts

- Repository: https://github.com/lyogavin/airllm
- Stars: 22,229 · Forks: 2,556 · Open issues: 106 · Watchers: 227
- Primary language: Jupyter Notebook
- License: Apache-2.0
- Last pushed: 2026-07-07T09:40:21+00:00

## Categories

- [Inference & Serving](/categories/inference-serving.md)
- [Model Training](/categories/model-training.md)

## Tags

llama, chinese-llm, llm, instruct-gpt, generative-ai, instruction-set, finetune, chinese-nlp

## Related tools

- [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models (★ 162,350)
- [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 144,582)
- [LLMs-from-scratch](/tools/rasbt-llms-from-scratch.md) - Implement a ChatGPT-like LLM in PyTorch from scratch (★ 98,715)
- [vllm](/tools/vllm-project-vllm.md) - A high-throughput and memory-efficient inference and serving engine for LLMs (★ 85,621)
- [rtk](/tools/rtk-ai-rtk.md) - CLI proxy reducing LLM token consumption by 60-90% (★ 69,277)
- [unsloth](/tools/unslothai-unsloth.md) - Unsloth Studio is a web UI for training and running open models locally. (★ 67,888)
- [anything-llm](/tools/mintplex-labs-anything-llm.md) - Stop renting your intelligence. Own it with AnythingLLM. (★ 62,770)
- [mem0](/tools/mem0ai-mem0.md) - Universal memory layer for AI Agents (★ 60,324)

## README (excerpt)

```text
[**Quickstart**](#quickstart) | 
[**Configurations**](#configurations) | 
[**MacOS**](#macos) | 
[**Example notebooks**](#example-python-notebook) | 
[**FAQ**](#faq)

**AirLLM** dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run **405B Llama 3.1** on **8GB**, and **DeepSeek-V3 (671B)** on **~12GB**.

<a href="https://github.com/lyogavin/airllm/stargazers"></a>





[
](https://pypi.org/project/airllm/)





## AI Agents Recommendation:

* [Best AI Game Sprite Generator](https://godmodeai.co)

* [Best AI Facial Expression Editor](https://crazyfaceai.com)

* [Bloome — build & run AI agent teams in the cloud, zero setup](https://bloome.im/login?ref=G6BYnov0)

## Updates
[2026/06] **v3.0**: FP8 model support + the latest models. Run **DeepSeek-V3 (671B) on ~12GB** and **Qwen3-235B on ~3GB**, plus Qwen3, Llama 3.x/4, DeepSeek V2/V3, Phi-4, Gemma and more — all through a single `AutoModel`.

[2024/08/20] v2.11.0: Support Qwen2.5

[2024/08/18] v2.10.1 Support CPU inference. Support non sharded models. Thanks @NavodPeiris for the great work! 

[2024/07/30] Support Llama3.1 **405B** ([example notebook](https://colab.research.google.com/github/lyogavin/airllm/blob/main/air_llm/examples/run_llama3.1_405B.ipynb)). Support **8bit/4bit quantization**.

[2024/04/20] AirLLM supports Llama3 natively already. Run Llama3 70B on 4GB single GPU.

[2023/12/25] v2.8.2: Support MacOS running 70B large language models.

[2023/12/20] v2.7: Support AirLLMMixtral. 

[2023/12/20] v2.6: Added AutoModel, automatically detect model type, no need to provide model class to initialize model.

[2023/12/18] v2.5: added prefetching to overlap the model loading and compute. 10% speed improvement.

[2023/12/03] added support of **ChatGLM**, **QWen**, **Baichuan**, **Mistral**, **InternLM**!

[2023/12/02] added support for safetensors. Now support all top 10 models in open llm leaderboard.

[2023/12/01] airllm 2.0. Support compressions: **3x run time speed up!**

[2023/11/20] airllm Initial version!

## Star History

<a href="https://star-history.com/#lyogavin/airllm&Timeline">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="assets/star-history-dark.png">
    <img alt="Star History Chart" src="assets/star-history.png">
  </picture>
</a>

## Table of Contents

* [Quick start](#quickstart)
* [Model Compression](#model-compression---3x-inference-speed-up)
* [Configurations](#configurations)
* [Run on MacOS](#macos)
* [Example notebooks](#example-python-notebook)
* [Supported Models](#supported-models)
* [Acknowledgement](#acknowledgement)
* [FAQ](#faq)

## Quickstart

### 1. Install package

First, install the airllm pip package.

```bash
pip install airllm
```

### 2. Inference

Then, initialize AirLLMLlama2, pass in the huggingface repo ID of the model being used, or the local path, and inference can be performed similar to a regular transformer model.

(*You can also specify the path to save the splitted layered model through **layer_shards_saving_path** when init AirLLMLlama2.*

```python
from airllm import AutoModel

MAX_LENGTH = 128
# just pass a hugging face repo id — works with almost any popular model:
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

# go bigger with the exact same one line:
#model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B")     # 235B, runs in ~3GB
#model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3")  # 671B, runs in ~12GB

# or use a model's local path...
#model = AutoModel.from_pretrained("/home/ubuntu/.cache/huggingface/hub/models--Qwen--Qwen3-32B/snapshots/...")

input_text = [
        'What is the capital of United States?',
        #'I like',
    ]

input_tokens = model.tokenizer(input_text,
    return_tensors="pt", 
    return_attention_mask=False, 
    truncation=True, 
    max_length=MAX_LENGTH, 
    padding=False)
           
generation_output = model.generate(
    input
```

---

**Machine-readable endpoints**

- JSON: [`/api/graphcanon/tools/lyogavin-airllm`](/api/graphcanon/tools/lyogavin-airllm)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)

_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_