{"data":{"slug":"staghado-vit-cpp","name":"vit.cpp","tagline":"Inference Vision Transformer in C/C++ with ggml","github_url":"https://github.com/staghado/vit.cpp","owner":"staghado","repo":"vit.cpp","owner_avatar_url":"https://avatars.githubusercontent.com/u/84044788?v=4","primary_language":"C++","stars":318,"forks":28,"topics":["ai","c","computer-vision","cpp","cpu","edge-computing","ggml","image-classification","llamacpp","vision-transformer","whisper-cpp"],"archived":false,"github_pushed_at":"2024-04-11T07:35:06+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/staghado-vit-cpp","markdown_url":"https://www.graphcanon.com/tools/staghado-vit-cpp.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/staghado-vit-cpp","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=staghado-vit-cpp","description":"Inference Vision Transformer (ViT) in plain C/C++ with ggml","homepage_url":null,"license":"MIT","open_issues":9,"watchers":10,"ai_summary":"A standalone implementation of the Vision Transformer (ViT) model for inference tasks using plain C/C++ and ggml.","readme_excerpt":"# vit.cpp\n\nInference Vision Transformer (ViT) in plain C/C++ using ggml without any extra dependencies\n\n## Description\n\n\nThis project presents a standalone implementation of the well known Vision Transformer (ViT) model family, used in a broad spectrum of applications and SOTA models like Large Multimodal Models(LMM). The primary goal is to develop a C/C++ inference engine tailored for ViT models, utilizing [ggml](https://github.com/ggerganov/ggml) to enhance performance, particularly on edge devices. Designed to be both lightweight and self-contained, this implementation can be run across diverse platforms.\n\n<details>\n<summary>Table of Contents</summary>\n\n1. [Description](#Description)\n2. [Features](#features)\n3. [Vision Transformer Architecture](#vision-transformer-architecture)\n4. [Quick Example](#quick-example)\n5. [Convert PyTorch to GGUF](#convert-pytorch-to-gguf)\n6. [Build](#build)\n   - [Simple Build](#simple-build)\n   - [Per Device Optimizations](#per-device-optimizations)\n   - [OpenMP](#using-openmp)\n7. [Run](#run)\n8. [Benchmark against PyTorch](#benchmark-against-pytorch)\n   - [ViT Inference](#vit-inference)\n   - [Benchmark on Your Machine](#benchmark-on-your-machine)\n9. [Quantization](#quantization)\n10. [To-Do List](#to-do-list)\n</details>\n\n\n## Features\n\n\n- Dependency-free and lightweight inference thanks to [ggml](https://github.com/ggerganov/ggml).\n- 4-bit, 5-bit and 8-bit quantization support.\n- Support for timm ViTs with different variants out of the box.\n\nAn important aspect of using `vit.cpp` is that it has short startup times compared to common DL frameworks, which makes it suitable for serverless deployments where the cold start is an issue.\n\n## Vision Transformer architecture\n\n\nThe implemented architecture is based on the original Vision Transformer from:\n  - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/abs/2010.11929)\n\n<p align=\"center\">\n  <img src=\"assets/image.png\" alt=\"Vision Transformer overview\" width=\"60%\" height=\"auto\">\n</p>\n<p align=\"center\">\n  ViT architecture. Taken from the <a href=\"https://arxiv.org/abs/2010.11929\">original paper</a>.\n</p>\n\n## Quick example\n\n\n<details>\n  <p align=\"center\">\n    <img src=\"assets/magpie.jpeg\" alt=\"example input\" width=\"50%\" height=\"auto\">\n  </p>\n\n  <summary>See output</summary>\n  <pre>\n  $ ./bin/vit -t 4 -m ../ggml-model-f16.gguf -i ../assets/magpie.jpeg -k 5\n  main: seed = 1701176263\n  main: n_threads = 4 / 8\n  vit_model_load: loading model from &apos;../ggml-model-f16.gguf&apos; - please wait\n  vit_model_load: hidden_size            = 192\n  vit_model_load: num_hidden_layers      = 12\n  vit_model_load: num_attention_heads    = 3\n  vit_model_load: patch_size             = 16\n  vit_model_load: img_size               = 224\n  vit_model_load: num_classes            = 1000\n  vit_model_load: ftype                  = 1\n  vit_model_load: qntvr                  = 0\n  operator(): ggml ctx size =  11.13 MB\n  vit_model_load: ................... done\n  vit_model_load: model size =    11.04 MB / num tensors = 152\n  main: loaded image &apos;../assets/magpie.jpeg&apos; (500 x 470)\n  vit_image_preprocess: scale = 2.232143\n  processed, out dims : (224 x 224)\n\n\n  &gt; magpie : 0.87\n  &gt; goose : 0.02\n  &gt; toucan : 0.01\n  &gt; drake : 0.01\n  &gt; king penguin, Aptenodytes patagonica : 0.01\n\n\n  main:    model load time =    17.92 ms\n  main:    processing time =   146.96 ms\n  main:    total time      =   164.88 ms\n  </pre>\n</details>\n\n\n## Convert PyTorch to GGUF\n\n\n    # clone the repo recursively\n    git clone --recurse-submodules https://github.com/staghado/vit.cpp.git\n\n    cd vit.cpp\n\n    # install torch and timm\n    pip install torch timm\n\n    # list available models if needed; note that not all models are supported\n    python convert-pth-to-ggml.py --list\n\n    # convert the weights to gguf : vit tiny with patch size of 16 and an image \n    # size of 384 pre-trained on ImageNet21k and fine-tuned on ImageNet1k\n    python convert-pth","github_created_at":"2023-11-02T14:38:02+00:00","created_at":"2026-07-11T12:31:31.225946+00:00","updated_at":"2026-08-01T00:00:44.501792+00:00","categories":[{"slug":"computer-vision","name":"Computer Vision","url":"https://www.graphcanon.com/categories/computer-vision","markdown_url":"https://www.graphcanon.com/categories/computer-vision.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/computer-vision"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"ai","name":"ai"},{"slug":"c","name":"c++"},{"slug":"computer-vision","name":"computer-vision"},{"slug":"cpp","name":"cpp"},{"slug":"cpu","name":"cpu"},{"slug":"edge-computing","name":"edge-computing"},{"slug":"ggml","name":"ggml"},{"slug":"image-classification","name":"image-classification"}],"trust":{"provenance":{"is_fork":false,"github_id":713446585,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-01T00:00:43.764Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":841,"last_release_at":"2023-12-11T13:53:16Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T12:31:34.561Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-01T00:00:44.238Z"},"languages":{"value":["c++"],"source":"github.language","observed_at":"2026-08-01T00:00:44.238Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-08-01T00:00:44.238Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["Use vit.cpp when you need fast startup times for serverless deployments as it addresses cold start issues inherent in common deep learning frameworks.","Select this tool for edge computing tasks due to its efficient use of CPU resources and support for different quantization levels, including 4-bit precision."],"when_not_to_use":["Avoid using vit.cpp if you require GPU acceleration since it is primarily optimized for CPU performance with ggml.","Do not choose vit.cpp if your project depends on rich ecosystem features or libraries unavailable in this standalone implementation lacking extra dependencies."],"source":"enrich:decision_facts","observed_at":"2026-07-16T20:56:45.255Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"vit.cpp is an optimized C/C++ implementation for Vision Transformer inference that leverages ggml to enhance performance and maintain lightweight, dependency-free operation."}]}}