{"data":{"slug":"yassa9-qwen600","name":"qwen600","tagline":"CUDA-only inference engine for qwen3-0.6B model","github_url":"https://github.com/yassa9/qwen600","owner":"yassa9","repo":"qwen600","owner_avatar_url":"https://avatars.githubusercontent.com/u/69548206?v=4","primary_language":"Cuda","stars":560,"forks":47,"topics":["cuda","cuda-programming","gpu","llamacpp","llm","llm-inference","qwen","qwen3","transformer"],"archived":false,"github_pushed_at":"2025-09-08T18:28:59+00:00","maintenance_label":"Dormant","stars_delta_30d":4,"url":"https://www.graphcanon.com/tools/yassa9-qwen600","markdown_url":"https://www.graphcanon.com/tools/yassa9-qwen600.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/yassa9-qwen600","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=yassa9-qwen600","description":"Static suckless single batch CUDA-only qwen3-0.6B mini inference engine","homepage_url":null,"license":"MIT","open_issues":1,"watchers":2,"ai_summary":"A static single batch CUDA inference implementation of the Qwen3-0.6B mini model focusing on efficiency and integration with the llamacpp framework.","readme_excerpt":"# qwen600.cu\n\n
\n \n
\n\nWhile studying and practicing CUDA & GPGPU, thought why not make an inference engine from scratch ? So, chose [QWEN3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) model, small model than can run smoothly on my `RTX 3050 8GB` VRAM.\nMy intention was (and still) to build educational program to learn about LLMs & transformers while maintaining practice in CUDA programming.\n\nI'm introducing static mini inference engine for `QWEN3-0.6B` instruct model in `bf16`, where its benchmarking claims that it's faster than [llama.cpp](https://github.com/ggml-org/llama.cpp) by approximately `8.5%` & `hf with flash-attn` by `292%` in `tokens/sec`, *see benchmarks below*.\n\n---\n\nWhat does `qwen600` include:\n- single batch inference engine\n- static-constanted for compile-time optimization\n- all CUDA C/C++, no python dependencies (except for tokenizer setup)\n- minimal libraries (cuBLAS, CUB, std IO)\n- efficient memory pipeline: mmap, single GPU block, async copy\n- zero-cost pointer-based weight management on GPU\n\n---\n\n`qwen600` is inspired by:\n- [llama.cpp - ggml](https://github.com/ggml-org/llama.cpp)\n- [llama2.c - Andrej Karpathy](https://github.com/karpathy/llama2.c)\n- [LLMs-from-scratch - Sebastian Raschka](https://github.com/rasbt/LLMs-from-scratch)\n- [qwen3.c - Adrian Cable](https://github.com/adriancable/qwen3.c)\n\n
\n \n
\n\n## Design Philosophy\n\n- The design of `qwen600.cu` is heavily inspired by the [suckless philosophy](https://suckless.org/philosophy/). \n- The goal is to create a tool that is simple, minimalist, and highly performant by avoiding feature bloat and unnecessary abstractions. \n- Configuration is done directly in the source code `config.h` as much as possible, and dependencies are kept to an absolute minimum.\n\n## WANNA TRY ?!\n\n### Initial Setup\n\nFirst, you need to clone [QWEN3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B).\nThis is fantastic [hugging face doc blog](https://huggingface.co/docs/hub/en/repositories-getting-started) to start with cloning hf repos.\n\nthen as a safe approach, you locate the weights file (model.safetensors) and sha256sum:\n```bash\nsha256sum /\n```\nand output must be according to hf:\n```text\nf47f71177f32bcd101b7573ec9171e6a57f4f4d31148d38e382306f42996874b\n```\n\nAfter that:\n\n```bash\ngit clone https://github.com/yassa9/qwen600\ncd qwen600\n```\n\nAssume that downloaded hugging face dir is ``.\n\nWe convert the Hugging Face tokenizer into the format used by `qwen600`.\n\n```bash\npython export.py \n```\n\nThat gonna output some template files and most importantly: `tokenizer.bin`\n\n### Building qwen600\n\nNow we are ready to build !\nYou just want: \n- `CUDA` + `nvcc`\n- `cuBLAS` + `CUB` \n\n```bash\nmkdir build && cd build\ncmake .. && make -j$(nproc)\n```\nJust that simple, no other bulky libraries and dependencies to build.\n\n## Moment of Truth: Running the Model\n\nYou can see arguments manual by: \n\n```bash\n# you are now inside qwen600/build\n./qwen600\n```\n\nthe output be that manual:\n```aiignore\nusage: ./qwen600 [options]\nexample: ./qwen600 -r 1\nmodel directory must contain:\n - model.safetensors\n - tokenizer.bin\n - template_*.txt files\n\narguments:\n----------\n -r reasoning mode, 0 (default) = no thinking, 1 = thinking\n -s random seed, default\n -k k value in top-k sampling, default 20\n -t temperature in [0,inf], default 0.6\n -p p value in top-p (nucleus) sampling in [0,1], default 0.95\n -i input prompt\n -y system prompt in chat mode, default is none\n```\n\nFor example:\n\n```bash\n./qwen600 -r 1 -t 0.65 -p 0.9 -k 20\n```\n\nor simply going with defaults:\n\n```bash\n./qwen600 -r 1\n```\n\nBased on official hugging face [model card](https://huggingface.co/Qwen/Qwen3-0.6B), they advise","github_created_at":"2025-09-06T01:42:22+00:00","created_at":"2026-07-11T11:45:16.968334+00:00","updated_at":"2026-09-20T06:35:20.670437+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"cuda","name":"cuda"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"qwen3","name":"qwen3"},{"slug":"transformer","name":"transformer"}],"trust":{"provenance":{"is_fork":false,"github_id":1051432019,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-20T03:44:23.493Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":376,"last_release_at":null,"stars_delta_30d":4,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:45:18.179Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-20T03:44:24.489Z"},"languages":{"value":["cuda"],"source":"github.language","observed_at":"2026-09-20T03:44:24.489Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-09-20T03:44:24.489Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium","summary":"Free to use due to MIT licensing; premium support or services might be available but are not detailed here."},"requirements":{"notes":["Requires a CUDA-compatible GPU","Integration with llamacpp framework necessary"]},"constraints":{"pricing_model":"freemium"},"when_to_use":["When you require high-performance, GPU-accelerated inference specifically tailored for the Qwen3-0.6B model.","If your deployment environment is based on the llamacpp framework and you seek seamless integration without additional configuration."],"when_not_to_use":["Avoid using when your hardware does not support CUDA or if you are running environments without access to compatible NVIDIA GPUs.","Do not select this tool if you need cross-platform compatibility, as qwen600 is strictly bound to CUDA and lacks functionality on non-CUDA systems."],"source":"enrich:decision_facts","observed_at":"2026-07-16T20:21:58.484Z"},"constraint_facets":{"pricing_model":"freemium"},"decision_summary":[{"label":"Pricing","value":"freemium - Free to use due to MIT licensing; premium support or services might be available but are not detailed here."},{"label":"Requirements","value":"Requires a CUDA-compatible GPU; Integration with llamacpp framework necessary"},{"label":"Adopt for","value":"qwen600 is a CUDA-exclusive inference engine designed to integrate with llamacpp for efficient performance of the Qwen3-0.6B model."},{"label":"License detail","value":"MIT license allows for free use, modification and distribution of the software."}]}}
![\"banner_\"](\"assets/banner.png\")
\n![\"arch\"](\"assets/arch.png\")
\n