qwen600.cu

While studying and practicing CUDA & GPGPU, thought why not make an inference engine from scratch ? So, chose QWEN3-0.6B model, small model than can run smoothly on my RTX 3050 8GB VRAM. My intention was (and still) to build educational program to learn about LLMs & transformers while maintaining practice in CUDA programming.

I'm introducing static mini inference engine for QWEN3-0.6B instruct model in bf16, where its benchmarking claims that it's faster than llama.cpp by approximately 8.5% & hf with flash-attn by 292% in tokens/sec, see benchmarks below.

What does qwen600 include:

• single batch inference engine
• static-constanted for compile-time optimization
• all CUDA C/C++, no python dependencies (except for tokenizer setup)
• minimal libraries (cuBLAS, CUB, std IO)
• efficient memory pipeline: mmap, single GPU block, async copy
• zero-cost pointer-based weight management on GPU

qwen600 is inspired by:

• llama.cpp - ggml
• llama2.c - Andrej Karpathy
• LLMs-from-scratch - Sebastian Raschka
• qwen3.c - Adrian Cable

Design Philosophy

• The design of qwen600.cu is heavily inspired by the suckless philosophy.
• The goal is to create a tool that is simple, minimalist, and highly performant by avoiding feature bloat and unnecessary abstractions.
• Configuration is done directly in the source code config.h as much as possible, and dependencies are kept to an absolute minimum.

WANNA TRY ?!

Initial Setup

First, you need to clone QWEN3-0.6B. This is fantastic hugging face doc blog to start with cloning hf repos.

then as a safe approach, you locate the weights file (model.safetensors) and sha256sum:

sha256sum <model_dir>/<safetensors-file-name>

and output must be according to hf:

f47f71177f32bcd101b7573ec9171e6a57f4f4d31148d38e382306f42996874b

After that:

git clone https://github.com/yassa9/qwen600
cd qwen600

Assume that downloaded hugging face dir is <model_dir>.

We convert the Hugging Face tokenizer into the format used by qwen600.

python export.py <model_dir>

That gonna output some template files and most importantly: tokenizer.bin

Building qwen600

Now we are ready to build ! You just want:

• CUDA + nvcc
• cuBLAS + CUB

mkdir build && cd build
cmake .. && make -j$(nproc)

Just that simple, no other bulky libraries and dependencies to build.

Moment of Truth: Running the Model

You can see arguments manual by:

# you are now inside qwen600/build
./qwen600

the output be that manual:

usage:   ./qwen600 <model_dir> [options]
example: ./qwen600 <model_dir> -r 1
model directory must contain:
  - model.safetensors
  - tokenizer.bin
  - template_*.txt files

arguments:
----------
  -r <int>    reasoning mode, 0 (default) = no thinking, 1 = thinking
  -s <int>    random seed, default
  -k <int>    k value in top-k sampling, default 20
  -t <float>  temperature in [0,inf], default 0.6
  -p <float>  p value in top-p (nucleus) sampling in [0,1], default 0.95
  -i <string> input prompt
  -y <string> system prompt in chat mode, default is none

For example:

./qwen600 <model_dir> -r 1 -t 0.65 -p 0.9 -k 20

or simply going with defaults:

./qwen600 <model_dir> -r 1

Based on official hugging face model card, they advise