---
title: "PowerInfer"
type: "tool"
slug: "tiiny-ai-powerinfer"
canonical_url: "https://www.graphcanon.com/tools/tiiny-ai-powerinfer"
github_url: "https://github.com/Tiiny-AI/PowerInfer"
homepage_url: null
stars: 9628
forks: 585
primary_language: "C++"
license: "MIT"
categories: ["inference-serving"]
tags: ["local-inference", "large-language-models", "llm-inference"]
updated_at: "2026-07-07T18:34:55.060331+00:00"
---
# PowerInfer
> High-speed Local LLM Serving
PowerInfer is a high-performance, low-resource inference engine designed for local deployment of Large Language Models (LLMs). It supports both CPU and GPU devices to leverage activation locality.
## Facts
- Repository: https://github.com/Tiiny-AI/PowerInfer
- Stars: 9,628 · Forks: 585 · Open issues: 129 · Watchers: 103
- Primary language: C++
- License: MIT
- Last pushed: 2026-05-11T06:48:06+00:00
## Categories
- [Inference & Serving](/categories/inference-serving.md)
## Tags
local-inference, large-language-models, llm-inference
## Related tools
- [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 144,575)
- [vllm](/tools/vllm-project-vllm.md) - A high-throughput and memory-efficient inference and serving engine for LLMs (★ 85,612)
- [rtk](/tools/rtk-ai-rtk.md) - CLI proxy reducing LLM token consumption by 60-90% (★ 69,253)
- [unsloth](/tools/unslothai-unsloth.md) - Unsloth Studio is a web UI for training and running open models locally. (★ 67,883)
- [anything-llm](/tools/mintplex-labs-anything-llm.md) - Stop renting your intelligence. Own it with AnythingLLM. (★ 62,759)
- [mem0](/tools/mem0ai-mem0.md) - Universal memory layer for AI Agents (★ 60,317)
- [litellm](/tools/berriai-litellm.md) - Python SDK and Proxy Server for calling over 100 LLM APIs in OpenAI format (★ 52,877)
- [ray](/tools/ray-project-ray.md) - Unified framework for scaling AI and Python applications (★ 43,153)
## README (excerpt)
```text
# PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
## TL;DR
PowerInfer is a CPU/GPU LLM inference engine leveraging **activation locality** for your device.
[Project Kanban](https://github.com/orgs/SJTU-IPADS/projects/2/views/2)
## Latest News 🔥
- [2026/1/5] We released **[Tiiny AI Pocket Lab](https://tiiny.ai/)**, the world's first pocket-size supercomputer. It runs GPT-OSS-120B (int4) locally at **20 tokens/s**. Featured at CES 2026.
- [2025/7/27] We released [SmallThinker-21BA3B-Instruct](https://huggingface.co/PowerInfer/SmallThinker-21BA3B-Instruct) and [SmallThinker-4BA0.6B-Instruct](https://huggingface.co/PowerInfer/SmallThinker-4BA0.6B-Instruct). We also released a corresponding framework for efficient [on-device inference](./smallthinker/README.md).
- [2024/6/11] We are thrilled to introduce [PowerInfer-2](https://arxiv.org/abs/2406.06282), our highly optimized inference framework designed specifically for smartphones. With TurboSparse-Mixtral-47B, it achieves an impressive speed of 11.68 tokens per second, which is up to 22 times faster than other state-of-the-art frameworks.
- [2024/6/11] We are thrilled to present [Turbo Sparse](https://arxiv.org/abs/2406.05955), our TurboSparse models for fast inference. With just $0.1M, we sparsified the original Mistral and Mixtral model to nearly 90% sparsity while maintaining superior performance! For a Mixtral-level model, our TurboSparse-Mixtral activates only **4B** parameters!
- [2024/5/20] **Competition Recruitment: CCF-TCArch Customized Computing Challenge 2024**. The CCF TCARCH CCC is a national competition organized by the Technical Committee on Computer Architecture (TCARCH) of the China Computer Federation (CCF). This year's competition aims to optimize the PowerInfer inference engine using the open-source ROCm/HIP. More information about the competition can be found [here](https://ccf-tcarch-ccc.github.io/2024/).
- [2024/5/17] We now provide support for AMD devices with ROCm.
- [2024/3/28] We are trilled to present [Bamboo LLM](https://github.com/SJTU-IPADS/Bamboo) that achieves both top-level performance and unparalleled speed with PowerInfer! Experience it with Bamboo-7B [Base](https://huggingface.co/PowerInfer/Bamboo-base-v0.1-gguf) / [DPO](https://huggingface.co/PowerInfer/Bamboo-DPO-v0.1-gguf).
- [2024/3/14] We supported ProSparse Llama 2 ([7B](https://huggingface.co/SparseLLM/prosparse-llama-2-7b)/[13B](https://huggingface.co/SparseLLM/prosparse-llama-2-13b)), ReLU models with ~90% sparsity, matching original Llama 2's performance (Thanks THUNLP & ModelBest)!
- [2024/1/11] We supported Windows with GPU inference!
- [2023/12/24] We released an online [gradio demo](https://powerinfer-gradio.vercel.app/) for Falcon(ReLU)-40B-FP16!
- [2023/12/19] We officially released PowerInfer!
## Demo 🔥
https://github.com/SJTU-IPADS/PowerInfer/assets/34213478/fe441a42-5fce-448b-a3e5-ea4abb43ba23
PowerInfer v.s. llama.cpp on a single RTX 4090(24G) running Falcon(ReLU)-40B-FP16 with a 11x speedup!
Both PowerInfer and llama.cpp were running on the same hardware and fully utilized VRAM on RTX 4090.
> [!NOTE]
> **Live Demo Online⚡️**
>
> Try out our [Gradio server](https://powerinfer-gradio.vercel.app/) hosting Falcon(ReLU)-40B-FP16 on a RTX 4090!
>
> Experimental and without warranties 🚧
## Abstract
We introduce PowerInfer, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC)
equipped with a single consumer-grade GPU. The key underlying the design of PowerInfer is exploiting the high **locality**
inherent in LLM inference, characterized by a power-law distribution in neuron activation.
This distribution indicates that a small subset of neurons, termed hot
```
---
**Machine-readable endpoints**
- JSON: [`/api/graphcanon/tools/tiiny-ai-powerinfer`](/api/graphcanon/tools/tiiny-ai-powerinfer)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)
_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_