Home/Inference & Serving/infinity

infinity

michaelfeil/infinity

Serving engine for text-embeddings, reranking models

GitHub Website

GraphCanon updated today · GitHub synced today

2.9k

Stars

196

Forks

130

Open issues

Watchers

3mo

Last push

Python MITCreated Oct 11, 2023

Trust & integrity

Full report

Maintenance: Slowing (109d since push); As of today · Source: github_public_v1
Provenance: Not a fork · Personal account; As of today · Source: github_public_v1
Security (OSV): No lockfile; As of today · Source: none

Public GitHub metadata and optional OSV dependency scans. Signals, not a guarantee. Trust methodology.

Overview

Infinity is a high-throughput, low-latency serving engine designed to serve text embeddings and other AI models including CLIP (Computer Vision model), CLAP (audio-text embedding model). It supports various models such as BERT embeddings via CLI or Docker.

Capability facts

Languages: python
Source: github.language · Jul 12, 2026

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Jul 11, 2026)

### Launch the cli via pip install

Source link

README

Launch the cli via pip install

pip install infinity-emb[all]

After your pip install, with your venv active, you can run the CLI directly.

infinity_emb v2 --model-id BAAI/bge-small-en-v1.5

Check the v2 --help command to get a description for all parameters.

infinity_emb v2 --help

Launch the CLI using a pre-built docker container (recommended)

Instead of installing the CLI via pip, you may also use docker to run michaelf34/infinity. Make sure you mount your accelerator ( i.e. install nvidia-docker and activate with --gpus all).

port=7997
model1=michaelfeil/bge-small-en-v1.5
model2=mixedbread-ai/mxbai-rerank-xsmall-v1
volume=$PWD/data

docker run -it --gpus all \
 -v $volume:/app/.cache \
 -p $port:$port \
 michaelf34/infinity:latest \
 v2 \
 --model-id $model1 \
 --model-id $model2 \
 --port $port

The cache path inside the docker container is set by the environment variable HF_HOME.

Specialized docker images

Docker container for CPU

Use the `latest-cpu` image or `x.x.x-cpu` for slimer image. Run like any other cpu-only docker image. Optimum/Onnx is often the prefered engine.

docker run -it \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest-cpu \
v2 \
--engine optimum \
--model-id $model1 \
--model-id $model2 \
--port $port

Docker Container for ROCm (MI200 Series and MI300 Series)

Use the `latest-rocm` image or `x.x.x-rocm` for rocm compatible inference. **This image is currently not build via CI/CD (to large), consider pinning to exact version.** Make sure you have ROCm is correctly installed and ready to use with Docker.

Visit Docs for more info.

Docker Container for Onnx-GPU, Cuda Extensions, TensorRT

Use the `latest-trt-onnx` image or `x.x.x-trt-onnx` for nvidia compatible inference. **This image is currently not build via CI/CD (to large), consider pinning to exact version.**

This image has support for:

ONNX-Cuda "CudaExecutionProvider"
ONNX-TensorRT "TensorRTExecutionProvider" (may not always work due to version mismatch with ORT)
CudaExtensions and packages, e.g. Tri-Dao's pip install flash-attn package when using Pytorch.
nvcc compiler support

docker run -it \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest-trt-onnx \
v2 \
--engine optimum \
--device cuda \
--model-id $model1 \
--port $port

Using local models with Docker container

In order to deploy a local model with a docker container, you need to mount the model inside the container and specify the path in the container to the launch command.

Example:

git lfs install 
cd /tmp
mkdir models && cd models && git clone https://huggingface.co/BAAI/bge-small-en-v1.5
docker run -it   -v /tmp/models:/models  -p 8081:8081  michaelf34/infinity:latest v2  --model-id "/models/bge-small-en-v1.5" --port 8081

Advanced CLI usage

Launching multiple models at once

Since infinity_emb>=0.0.34, you can use cli v2 method to launch multiple models at the same time. Checkout infinity_emb v2 --help for all args and validation.

Multiple Model CLI Playbook:

1. cli options can be repeated e.g. v2 --model-id model/id1 --model-id model/id2 --batch-size 8 --batch-size 4. This will create two models model/id1 and model/id2
1. or adapt the defaults by setting ENV Variables separated by ;: INFINITY_MODEL_ID="model/id1;model/id2;" && INFINITY_BATCH_SIZE="8;4;"
1. single items are broadcasted to --model-id length, v2 --model-id model/id1 --model-id/id2 --batch-size 8 making both models have batch-size 8.
1. Everythin