{"data":{"slug":"jmaczan-tiny-vllm","name":"tiny-vllm","tagline":"Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM","github_url":"https://github.com/jmaczan/tiny-vllm","owner":"jmaczan","repo":"tiny-vllm","owner_avatar_url":"https://avatars.githubusercontent.com/u/18054202?v=4","primary_language":"C++","stars":1120,"forks":92,"topics":["ai","attention","batching","course","cpp","cuda","hpc","inference","llm","llm-inference","pagedattention","tiny-vllm","vllm"],"archived":false,"github_pushed_at":"2026-09-15T19:46:09+00:00","maintenance_label":"Very active","stars_delta_30d":173,"url":"https://www.graphcanon.com/tools/jmaczan-tiny-vllm","markdown_url":"https://www.graphcanon.com/tools/jmaczan-tiny-vllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/jmaczan-tiny-vllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=jmaczan-tiny-vllm","description":"Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM","homepage_url":null,"license":"Apache-2.0","open_issues":1,"watchers":6,"ai_summary":"tiny-vllm is a C++ and CUDA-based framework for creating lightweight yet powerful large language model (LLM) inference engines, inspired by but scaled down from vLLM.","readme_excerpt":"# tiny-vllm\n\nYou're going to build a high performance LLM inference engine with C++ and CUDA - tiny-vllm, a younger and smaller sibling of [vLLM](https://github.com/vllm-project/vllm)\n\nWe will learn a lot along the way, make mistakes and derive the ideas and maths from scratch\n\nThis repository consists of two things: 1. a full source code of the inference server and 2. a course where I lead you through the process of implementing the engine. Feel invited to use it as a learning tool on your learning path or if you are a lecturer, feel welcome to use it as a teaching resource at your university\n\nThe inference engine consists of:\n\n- [x] load a real LLM model from Safetensors (Llama 3.2 1B Instruct)\n- [x] full LLM forward pass (prefill + decode)\n- [x] all computation with CUDA kernels\n- [x] KV cache\n- [x] static batching\n- [x] continuous batching\n- [x] [online softmax, FlashAttention-like](https://courses.cs.washington.edu/courses/cse599m/23sp/notes/flashattn.pdf)\n- [x] [PagedAttention](https://arxiv.org/pdf/2309.06180)\n\nMake yourself a hot beverage and let's begin\n\n- [tiny-vllm](#tiny-vllm)\n  - [Intro: LLM, vLLM, models, inference servers](#intro-llm-vllm-models-inference-servers)\n  - [Technical prerequisities](#technical-prerequisities)\n  - [Safetensors and your model](#safetensors-and-your-model)\n  - [How floating-point numbers work and why we use bfloat16](#how-floating-point-numbers-work-and-why-we-use-bfloat16)\n  - [GPU and CPU memory](#gpu-and-cpu-memory)\n  - [Single token inference](#single-token-inference)\n  - [Tokenization](#tokenization)\n  - [Embeddings](#embeddings)\n  - [CUDA kernel engineering - embeddings](#cuda-kernel-engineering---embeddings)\n  - [RMSNorm and parallel reduction in CUDA](#rmsnorm-and-parallel-reduction-in-cuda)\n  - [RoPE](#rope)\n  - [Residual connections](#residual-connections)\n  - [cublasGemmEx](#cublasgemmex)\n  - [The column-major to row-major transposition trick](#the-column-major-to-row-major-transposition-trick)\n  - [Prefill vs decode](#prefill-vs-decode)\n  - [Why KV cache exists](#why-kv-cache-exists)\n  - [Attention](#attention)\n  - [GQA](#gqa)\n  - [SiLU](#silu)\n  - [Softmax](#softmax)\n  - [Causal mask](#causal-mask)\n  - [Argmax](#argmax)\n  - [Feed forward network](#feed-forward-network)\n  - [Buffer reuse](#buffer-reuse)\n  - [Static batching](#static-batching)\n  - [Continuous batching](#continuous-batching)\n  - [Online softmax](#online-softmax)\n  - [Paged Attention](#paged-attention)\n  - [Paged KV cache](#paged-kv-cache)\n  - [Paged Attention CUDA kernel](#paged-attention-cuda-kernel)\n\n## Intro: LLM, vLLM, models, inference servers\n\nIt's easy to get lost with so much going on recent years. Let's unpack it\n\n**LLM is a model**. Physically, **LLM is a file which contains a lot of [float numbers](https://en.wikipedia.org/wiki/Floating-point_arithmetic)**. Conceptually, these numbers represent weights of operations. Weights are learned/discovered/found during training phase. Some of the operations use these weights. Every operation is a function, which takes some data as input, do something with it and produces data as output. Operations and their order are defined by LLM's architecture. Every model has its own architecture, which is designed by engineers and researchers.\n\nThe process of going from 0 to LLM writing a text is like this:\n\n0. **Design the model** - engineers and researchers use high level language like Python with tensor library like [PyTorch](https://github.com/pytorch/pytorch) or [tinygrad](https://github.com/tinygrad/tinygrad) to design model's architecture. They train small versions of the model, make experiments with different operations, data and hyperparameters (parameters for operations). It's the phase of figuring out the specification\n1. **Implement the model** - Once they decide on final model architecture and prepare the data for training, they write the code that defines the final model. It can be also in PyTorch or similar\n2. **Train the model** - The chosen model archit","github_created_at":"2026-02-09T16:40:15+00:00","created_at":"2026-07-11T11:44:10.840335+00:00","updated_at":"2026-09-20T06:34:17.439708+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"cuda","name":"cuda"},{"slug":"hpc","name":"hpc"},{"slug":"llm","name":"llm"},{"slug":"lstm","name":"lstm"}],"trust":{"provenance":{"is_fork":false,"github_id":1153764295,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-19T18:31:57.289Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":3,"last_release_at":null,"stars_delta_30d":173,"open_issues_delta_30d":-1},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:44:11.972Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-19T18:31:58.291Z"},"languages":{"value":["c++"],"source":"github.language","observed_at":"2026-09-19T18:31:58.291Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-19T18:31:58.291Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you require a lightweight solution for deploying large language model inference in environments with limited resources but still demand high performance.","If the infrastructure supports C++ and CUDA and there is a need to develop custom solutions without the overhead of vLLM."],"when_not_to_use":["Avoid using tiny-vllm if the application requires the full feature set offered by its larger counterpart, vLLM, as it has been trimmed for lightweight use.","Do not choose this tool when working in environments that do not support CUDA or where a higher abstraction level is preferred over direct C++ and CUDA implementation."],"source":"enrich:decision_facts","observed_at":"2026-07-15T09:59:07.004Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"For those needing a compact yet potent LLM inference engine built on C++ and CUDA, tiny-vllm presents an accessible framework inspired by its larger sibling, vLLM."}]}}