tiny-vllm
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026
57views this month
Decision brief
For those needing a compact yet potent LLM inference engine built on C++ and CUDA, tiny-vllm presents an accessible framework inspired by its larger sibling, vLLM.
Good fit when
- When you require a lightweight solution for deploying large language model inference in environments with limited resources but still demand high performance.
- If the infrastructure supports C++ and CUDA and there is a need to develop custom solutions without the overhead of vLLM.
Avoid when
- Avoid using tiny-vllm if the application requires the full feature set offered by its larger counterpart, vLLM, as it has been trimmed for lightweight use.
- Do not choose this tool when working in environments that do not support CUDA or where a higher abstraction level is preferred over direct C++ and CUDA implementation.
Observed Jul 15, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Very active (3d since push)
- As of Sep 19, 2026
- Provenance
- Not a fork · Personal account
- As of Sep 19, 2026
- Security (OSV)
- No lockfile
- As of Jul 11, 2026
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
git clone https://github.com/jmaczan/tiny-vllmHow it fits your stack(1)
Typed graph edges - alternatives, integrations, successors, and dependencies. Ranked by relationship type, not raw GitHub stars.
Relationship graph
Optional deeper exploration of typed edges and category neighbours.
Similar tools
Same-category neighbours not already linked as typed edges.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
tiny-vllm is a C++ and CUDA-based framework for creating lightweight yet powerful large language model (LLM) inference engines, inspired by but scaled down from vLLM.
Capability facts
- Languages
- c++
Source: github.language · Sep 19, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Sep 19, 2026)
. **Design the model** - engineers and researchers use high level language like Python with tensor library like [PyTorch](https://github.com/pytorch/pytorch) or [tinygSource link
Tags
README
tiny vllm You're going to build a high performance LLM inference engine with C++ and CUDA tiny vllm, a younger and smaller sibling of vLLM We will learn a lot along the way, make mistakes and derive the ideas and maths from scratch This repository consists of two things: 1. a ful...
For agents
This page has a .md twin and JSON over the API.