PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

TL;DR

PowerInfer is a CPU/GPU LLM inference engine leveraging activation locality for your device.

Project Kanban

Latest News 🔥

• [2026/1/5] We released Tiiny AI Pocket Lab, the world's first pocket-size supercomputer. It runs GPT-OSS-120B (int4) locally at 20 tokens/s. Featured at CES 2026.
• [2025/7/27] We released SmallThinker-21BA3B-Instruct and SmallThinker-4BA0.6B-Instruct. We also released a corresponding framework for efficient on-device inference.
• [2024/6/11] We are thrilled to introduce PowerInfer-2, our highly optimized inference framework designed specifically for smartphones. With TurboSparse-Mixtral-47B, it achieves an impressive speed of 11.68 tokens per second, which is up to 22 times faster than other state-of-the-art frameworks.
• [2024/6/11] We are thrilled to present Turbo Sparse, our TurboSparse models for fast inference. With just $0.1M, we sparsified the original Mistral and Mixtral model to nearly 90% sparsity while maintaining superior performance! For a Mixtral-level model, our TurboSparse-Mixtral activates only 4B parameters!
• [2024/5/20] Competition Recruitment: CCF-TCArch Customized Computing Challenge 2024. The CCF TCARCH CCC is a national competition organized by the Technical Committee on Computer Architecture (TCARCH) of the China Computer Federation (CCF). This year's competition aims to optimize the PowerInfer inference engine using the open-source ROCm/HIP. More information about the competition can be found here.
• [2024/5/17] We now provide support for AMD devices with ROCm.
• [2024/3/28] We are trilled to present Bamboo LLM that achieves both top-level performance and unparalleled speed with PowerInfer! Experience it with Bamboo-7B Base / DPO.
• [2024/3/14] We supported ProSparse Llama 2 (7B/13B), ReLU models with ~90% sparsity, matching original Llama 2's performance (Thanks THUNLP & ModelBest)!
• [2024/1/11] We supported Windows with GPU inference!
• [2023/12/24] We released an online gradio demo for Falcon(ReLU)-40B-FP16!
• [2023/12/19] We officially released PowerInfer!

Demo 🔥

https://github.com/SJTU-IPADS/PowerInfer/assets/34213478/fe441a42-5fce-448b-a3e5-ea4abb43ba23

PowerInfer v.s. llama.cpp on a single RTX 4090(24G) running Falcon(ReLU)-40B-FP16 with a 11x speedup!

_{Both PowerInfer and llama.cpp were running on the same hardware and fully utilized VRAM on RTX 4090.}

[!NOTE] Live Demo Online⚡️

Try out our Gradio server hosting Falcon(ReLU)-40B-FP16 on a RTX 4090!

_{Experimental and without warranties 🚧}

Abstract

We introduce PowerInfer, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key underlying the design of PowerInfer is exploiting the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation.

This distribution indicates that a small subset of neurons, termed hot