2. Install (uses the upstream precompiled wheel; KVarN kernels are Triton, JIT-compiled at runtime)

VLLM_USE_PRECOMPILED=1 pip install -e .

```python
from vllm import LLM, SamplingParams

llm = LLM(
    model="Qwen/Qwen3-32B",
    dtype="float16",                    # KVarN runs in float16
    kv_cache_dtype="kvarn_k4v2_g128",   # enable KVarN
    block_size=128,                     # KVarN tile size
)
print(llm.generate("Explain KV-cache quantization in one sentence.",
                    SamplingParams(max_tokens=64))[0].outputs[0].text)

Serving works the same way:

vllm serve Qwen/Qwen3-32B --dtype float16 --kv-cache-dtype kvarn_k4v2_g128 --block-size 128

Note: KVarN runs in float16 compute. One vLLM block is one KVarN tile, so the tile / page size equals --block-size. Both 128 (default) and 64 are supported, selected by the matching preset (kvarn_k4v2_g128 / kvarn_k4v2_g64). 128 is the design point; 64 gives finer quantization granularity at the cost of a little KV capacity (more per-tile scale overhead per token), at essentially the same throughput.

Tip (capacity): KVarN realizes its full KV-cache capacity when there is room to amortize a small fixed decode workspace. On multi-GPU or generous --gpu-memory-utilization setups this is automatic. On a tight single-GPU budget, vLLM's CUDA-graph memory profiler can over-reserve and shrink the KV pool; set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 (and/or raise --gpu-memory-utilization) to recover the full capacity.

License and attribution

KVarN is built on vLLM (v0.23.0) and is released under the Apache 2.0 License. The original vLLM README is preserved as README_vLLM.md.