GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026
40views this month
Decision brief
quant.cpp, a lossless KV cache compression and quantization tool for LLM inference in pure C without dependencies.
Good fit when
- Use quant.cpp when you need extended context for LLM inference in a lightweight, embeddable environment with no external dependencies.
- Opt for quant.cpp if your scenario benefits from a single-header library that enables seamless integration into existing C codebases.
Avoid when
- Avoid using quant.cpp for projects requiring non-C language support or frameworks since it strictly operates within the context of pure C.
- Do not use quant.cpp in environments where rapid runtime performance is paramount and additional compile-time overhead introduced by its unique compression techniques may cause delays.
- Requirements:
- Requires a C compiler compatible with quant.cpp source code.
Observed Jul 16, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Slowing (146d since push)
- As of Sep 20, 2026
- Provenance
- Not a fork · Organization account
- As of Sep 20, 2026
- Security (OSV)
- No lockfile
- As of Jul 11, 2026
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
git clone https://github.com/quantumaikr/quant.cppSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
quantumaikr/quant.cpp offers lossless KV cache compression and quantization for LLM inference in pure C without dependencies.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Sep 20, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Sep 20, 2026
- Languages
- c
Source: github.language · Sep 20, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Sep 20, 2026)
**Python API (3 lines):**Source link
Source: README excerpt (regex_v1, Sep 20, 2026)
quantcpp serve qwen3 -p 8080 # OpenAI-compatible HTTP server (SSE streaming)Source link
Tags
README
Quick Start Ollama style CLI (v0.12.0+): Recommended default: Qwen3 4B (4B params, MMLU 73, 4.5 tok/s on M3). Best speed AND quality — the Q4 NEON fused dot path makes it 2.4x faster than Phi 3.5 mini despite a larger vocab. Other aliases: , , . Auto pulls on first / . The subcom...
For agents
This page has a .md twin and JSON over the API.