vit.cpp

Inference Vision Transformer (ViT) in plain C/C++ using ggml without any extra dependencies

Description

This project presents a standalone implementation of the well known Vision Transformer (ViT) model family, used in a broad spectrum of applications and SOTA models like Large Multimodal Models(LMM). The primary goal is to develop a C/C++ inference engine tailored for ViT models, utilizing ggml to enhance performance, particularly on edge devices. Designed to be both lightweight and self-contained, this implementation can be run across diverse platforms.

Features

• Dependency-free and lightweight inference thanks to ggml.
• 4-bit, 5-bit and 8-bit quantization support.
• Support for timm ViTs with different variants out of the box.

An important aspect of using vit.cpp is that it has short startup times compared to common DL frameworks, which makes it suitable for serverless deployments where the cold start is an issue.

Vision Transformer architecture

The implemented architecture is based on the original Vision Transformer from:

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

ViT architecture. Taken from the original paper.

Quick example

  $ ./bin/vit -t 4 -m ../ggml-model-f16.gguf -i ../assets/magpie.jpeg -k 5
  main: seed = 1701176263
  main: n_threads = 4 / 8
  vit_model_load: loading model from '../ggml-model-f16.gguf' - please wait
  vit_model_load: hidden_size            = 192
  vit_model_load: num_hidden_layers      = 12
  vit_model_load: num_attention_heads    = 3
  vit_model_load: patch_size             = 16
  vit_model_load: img_size               = 224
  vit_model_load: num_classes            = 1000
  vit_model_load: ftype                  = 1
  vit_model_load: qntvr                  = 0
  operator(): ggml ctx size =  11.13 MB
  vit_model_load: ................... done
  vit_model_load: model size =    11.04 MB / num tensors = 152
  main: loaded image '../assets/magpie.jpeg' (500 x 470)
  vit_image_preprocess: scale = 2.232143
  processed, out dims : (224 x 224)
> magpie : 0.87
> goose : 0.02
> toucan : 0.01
> drake : 0.01
> king penguin, Aptenodytes patagonica : 0.01
main:    model load time =    17.92 ms
main:    processing time =   146.96 ms
main:    total time      =   164.88 ms

Convert PyTorch to GGUF

# clone the repo recursively
git clone --recurse-submodules https://github.com/staghado/vit.cpp.git

cd vit.cpp

# install torch and timm
pip install torch timm

# list available models if needed; note that not all models are supported
python convert-pth-to-ggml.py --list

# convert the weights to gguf : vit tiny with patch size of 16 and an image 
# size of 384 pre-trained on ImageNet21k and fine-tuned on ImageNet1k
python convert-pth