{"data":{"slug":"huawei-csl-kvarn","name":"KVarN","tagline":"vLLM KV-cache quantization backend for AI agents","github_url":"https://github.com/huawei-csl/KVarN","owner":"huawei-csl","repo":"KVarN","owner_avatar_url":"https://avatars.githubusercontent.com/u/234563461?v=4","primary_language":"Python","stars":497,"forks":36,"topics":["agentic-ai","kv-cache","llm","llm-inference","long-context","quantization","vllm"],"archived":false,"github_pushed_at":"2026-06-22T09:18:28+00:00","maintenance_label":"Steady","stars_delta_30d":55,"url":"https://www.graphcanon.com/tools/huawei-csl-kvarn","markdown_url":"https://www.graphcanon.com/tools/huawei-csl-kvarn.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/huawei-csl-kvarn","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=huawei-csl-kvarn","description":"KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.","homepage_url":"https://arxiv.org/abs/2606.03458","license":"Apache-2.0","open_issues":11,"watchers":5,"ai_summary":"KVarN offers a quantization solution that enhances the context length and throughput of AI agents without compromising on accuracy.","readme_excerpt":"# 2. Install (uses the upstream precompiled wheel; KVarN kernels are Triton, JIT-compiled at runtime)\nVLLM_USE_PRECOMPILED=1 pip install -e .\n```\n\n```python\nfrom vllm import LLM, SamplingParams\n\nllm = LLM(\n    model=\"Qwen/Qwen3-32B\",\n    dtype=\"float16\",                    # KVarN runs in float16\n    kv_cache_dtype=\"kvarn_k4v2_g128\",   # enable KVarN\n    block_size=128,                     # KVarN tile size\n)\nprint(llm.generate(\"Explain KV-cache quantization in one sentence.\",\n                    SamplingParams(max_tokens=64))[0].outputs[0].text)\n```\n\nServing works the same way:\n\n```bash\nvllm serve Qwen/Qwen3-32B --dtype float16 --kv-cache-dtype kvarn_k4v2_g128 --block-size 128\n```\n\n> **Note:** KVarN runs in `float16` compute. One vLLM block is one KVarN tile, so\n> the tile / page size equals `--block-size`. Both **128** (default) and **64** are\n> supported, selected by the matching preset (`kvarn_k4v2_g128` / `kvarn_k4v2_g64`).\n> 128 is the design point; 64 gives finer quantization granularity at the cost of a\n> little KV capacity (more per-tile scale overhead per token), at essentially the\n> same throughput.\n\n> **Tip (capacity):** KVarN realizes its full KV-cache capacity when there is room\n> to amortize a small fixed decode workspace. On multi-GPU or generous\n> `--gpu-memory-utilization` setups this is automatic. On a tight single-GPU budget,\n> vLLM's CUDA-graph memory profiler can over-reserve and shrink the KV pool; set\n> `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` (and/or raise\n> `--gpu-memory-utilization`) to recover the full capacity.\n\n---\n\n## License and attribution\n\nKVarN is built on [vLLM](https://github.com/vllm-project/vllm) (v0.23.0) and is\nreleased under the Apache 2.0 License. The original vLLM README is preserved as\n[`README_vLLM.md`](README_vLLM.md).","github_created_at":"2026-05-29T08:19:15+00:00","created_at":"2026-07-11T11:46:25.155029+00:00","updated_at":"2026-09-20T06:36:27.714606+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"agentic-ai","name":"agentic-ai"},{"slug":"kv-cache","name":"kv-cache"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"long-context","name":"long-context"},{"slug":"quantization","name":"quantization"},{"slug":"vllm","name":"vllm"}],"trust":{"provenance":{"is_fork":false,"github_id":1253187784,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-20T03:46:10.050Z","maintenance":{"label":"Steady","score":60,"methodology":"github_public_v1","releases_90d":0,"days_since_push":89,"last_release_at":null,"stars_delta_30d":55,"open_issues_delta_30d":3},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:46:26.595Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-20T03:46:11.093Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-09-20T03:46:11.093Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-20T03:46:11.093Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-20T03:46:11.093Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["For applications needing over threefold to fivefold increase in context length compared to FP16.","When seamless integration with minimal setup, using one flag is desired."],"when_not_to_use":["If project constraints do not allow for Apache-2.0 licensing terms.","Projects that cannot benefit from a quantization backend, such as those requiring non-variable length model support."],"source":"enrich:decision_facts","observed_at":"2026-07-15T10:26:19.421Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"KVarN amplifies AI agent capabilities via vLLM KV-cache quantization for extended context and throughput without sacrificing accuracy."}]}}