GraphCanon updated today · GitHub synced today
Decision brief
quant.cpp, a lossless KV cache compression and quantization tool for LLM inference in pure C without dependencies.
Good fit when
- Use quant.cpp when you need extended context for LLM inference in a lightweight, embeddable environment with no external dependencies.
- Opt for quant.cpp if your scenario benefits from a single-header library that enables seamless integration into existing C codebases.
Avoid when
- Avoid using quant.cpp for projects requiring non-C language support or frameworks since it strictly operates within the context of pure C.
- Do not use quant.cpp in environments where rapid runtime performance is paramount and additional compile-time overhead introduced by its unique compression techniques may cause delays.
- Requirements:
- Requires a C compiler compatible with quant.cpp source code.
Observed Jul 16, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Slowing (121d since push)
- As of today
- Provenance
- Not a fork · Organization account
- As of today
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
git clone https://github.com/quantumaikr/quant.cppSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
quantumaikr/quant.cpp offers lossless KV cache compression and quantization for LLM inference in pure C without dependencies.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Aug 25, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Aug 25, 2026
- Languages
- c
Source: github.language · Aug 25, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 25, 2026)
**Python API (3 lines):**Source link
Source: README excerpt (regex_v1, Aug 25, 2026)
quantcpp serve qwen3 -p 8080 # OpenAI-compatible HTTP server (SSE streaming)Source link
Tags
README
Quick Start
Ollama-style CLI (v0.12.0+):
pip install quantcpp
quantcpp pull qwen3 # download Qwen3-4B Q4_K_M (~2.5 GB)
quantcpp run qwen3 # interactive chat
quantcpp serve qwen3 -p 8080 # OpenAI-compatible HTTP server (SSE streaming)
quantcpp client "Hi" # streaming client → server on :8080
quantcpp list # show cached models
Recommended default: Qwen3-4B (4B params, MMLU 73, 4.5 tok/s on M3). Best speed AND quality — the Q4 NEON fused dot path makes it 2.4x faster than Phi-3.5-mini despite a larger vocab. Other aliases: phi3.5, smollm2, llama3.2:1b. Auto-pulls on first run / serve.
The serve subcommand exposes POST /v1/chat/completions (OpenAI-compatible) on port 8080 — clients pass "stream": true for SSE streaming, or omit it for a single JSON response. Built-in quantcpp client supports both modes (default: streaming, --no-stream for single response).
One-shot question:
quantcpp run qwen3 "What is gravity?"
Python API (3 lines):
from quantcpp import Model
m = Model.from_pretrained("Qwen3-4B")
print(m.ask("What is gravity?"))
Downloads on first use, cached at ~/.cache/quantcpp/. No API key, no GPU. See docs/supported_models.md for the architecture support matrix and model selection guide. Try in browser → · Interactive Guide →
Pick any GGUF you have on disk (this is the one from Quick Start):
MODEL=models/SmolLM2-135M-Instruct-Q8_0.gguf
Docker & Server
Docker (zero-dependency, ~10MB image):
docker build -t quant.cpp .
docker run -v ./models:/models quant.cpp /models/SmolLM2-135M-Instruct-Q8_0.gguf -p "hello" -k uniform_4b -v q4
For agents
This page has a .md twin and JSON over the API.