{"data":{"slug":"quantumaikr-quant-cpp","name":"quant.cpp","tagline":"LLM inference with extended context using C","github_url":"https://github.com/quantumaikr/quant.cpp","owner":"quantumaikr","repo":"quant.cpp","owner_avatar_url":"https://avatars.githubusercontent.com/u/49489033?v=4","primary_language":"C","stars":399,"forks":44,"topics":["delta-compression","embeddable","gguf","kv-cache","llm","llm-inference","pure-c","quantization","transformer","turboquant"],"archived":false,"github_pushed_at":"2026-04-26T11:15:51+00:00","maintenance_label":"Slowing","stars_delta_30d":4,"url":"https://www.graphcanon.com/tools/quantumaikr-quant-cpp","markdown_url":"https://www.graphcanon.com/tools/quantumaikr-quant-cpp.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/quantumaikr-quant-cpp","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=quantumaikr-quant-cpp","description":"LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.","homepage_url":null,"license":"Apache-2.0","open_issues":11,"watchers":7,"ai_summary":"quantumaikr/quant.cpp offers lossless KV cache compression and quantization for LLM inference in pure C without dependencies.","readme_excerpt":"## Quick Start\n\n**Ollama-style CLI (v0.12.0+):**\n```bash\npip install quantcpp\n\nquantcpp pull qwen3                     # download Qwen3-4B Q4_K_M (~2.5 GB)\nquantcpp run qwen3                      # interactive chat\nquantcpp serve qwen3 -p 8080            # OpenAI-compatible HTTP server (SSE streaming)\nquantcpp client \"Hi\"                    # streaming client → server on :8080\nquantcpp list                           # show cached models\n```\n\nRecommended default: **Qwen3-4B** (4B params, MMLU 73, 4.5 tok/s on M3). Best speed AND quality — the Q4 NEON fused dot path makes it 2.4x faster than Phi-3.5-mini despite a larger vocab. Other aliases: `phi3.5`, `smollm2`, `llama3.2:1b`. Auto-pulls on first `run` / `serve`.\n\nThe `serve` subcommand exposes `POST /v1/chat/completions` (OpenAI-compatible) on port 8080 — clients pass `\"stream\": true` for SSE streaming, or omit it for a single JSON response. Built-in `quantcpp client` supports both modes (default: streaming, `--no-stream` for single response).\n\n**One-shot question:**\n```bash\nquantcpp run qwen3 \"What is gravity?\"\n```\n\n**Python API (3 lines):**\n```python\nfrom quantcpp import Model\nm = Model.from_pretrained(\"Qwen3-4B\")\nprint(m.ask(\"What is gravity?\"))\n```\n\nDownloads on first use, cached at `~/.cache/quantcpp/`. No API key, no GPU. See [`docs/supported_models.md`](docs/supported_models.md) for the architecture support matrix and model selection guide. [Try in browser →](https://quantumaikr.github.io/quant.cpp/) · [**Interactive Guide →**](https://quantumaikr.github.io/quant.cpp/guide/)\n\n---\n\n---\n\n# Pick any GGUF you have on disk (this is the one from Quick Start):\nMODEL=models/SmolLM2-135M-Instruct-Q8_0.gguf\n\n---\n\n## Docker & Server\n\n**Docker** (zero-dependency, ~10MB image):\n```bash\ndocker build -t quant.cpp .\ndocker run -v ./models:/models quant.cpp /models/SmolLM2-135M-Instruct-Q8_0.gguf -p \"hello\" -k uniform_4b -v q4","github_created_at":"2026-03-28T22:07:24+00:00","created_at":"2026-07-11T11:46:43.768828+00:00","updated_at":"2026-08-25T12:01:26.347082+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"delta-compression","name":"delta-compression"},{"slug":"embeddable","name":"embeddable"},{"slug":"gguf","name":"gguf"},{"slug":"kv-cache","name":"kv-cache"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"pure-c","name":"pure-c"},{"slug":"quantization","name":"quantization"},{"slug":"transformer","name":"transformer"}],"trust":{"provenance":{"is_fork":false,"github_id":1194842235,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-25T12:01:25.534Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":121,"last_release_at":"2026-04-12T10:54:46Z","stars_delta_30d":4,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:46:44.981Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-25T12:01:26.038Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-08-25T12:01:26.038Z","managed_saas":false},"languages":{"value":["c"],"source":"github.language","observed_at":"2026-08-25T12:01:26.038Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-08-25T12:01:26.038Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-25T12:01:26.038Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Requires a C compiler compatible with quant.cpp source code."],"requires_docker":false},"constraints":{"requires_docker":false},"when_to_use":["Use quant.cpp when you need extended context for LLM inference in a lightweight, embeddable environment with no external dependencies.","Opt for quant.cpp if your scenario benefits from a single-header library that enables seamless integration into existing C codebases."],"when_not_to_use":["Avoid using quant.cpp for projects requiring non-C language support or frameworks since it strictly operates within the context of pure C.","Do not use quant.cpp in environments where rapid runtime performance is paramount and additional compile-time overhead introduced by its unique compression techniques may cause delays."],"source":"enrich:decision_facts","observed_at":"2026-07-16T21:15:28.994Z"},"constraint_facets":{"requires_docker":false},"decision_summary":[{"label":"Requirements","value":"Requires a C compiler compatible with quant.cpp source code."},{"label":"Adopt for","value":"quant.cpp, a lossless KV cache compression and quantization tool for LLM inference in pure C without dependencies."},{"label":"License detail","value":"Quant.cpp uses the Apache-2.0 license, which allows for free use, modification, and distribution. Contributions to its codebase are welcomed."}]}}