Home/Inference & Serving/infinity

infinity

michaelfeil/infinity

High-throughput, low-latency serving engine for text-embeddings and various models

GitHub Website

GraphCanon updated 2w · GitHub synced 2w

2.9k stars196 forksLast push 5mo Python MIT

Decision brief

Infinity is a high-throughput, low-latency serving engine that supports text-embeddings, reranking models, CLIP, CLAP, and ColPaLi, with GPU acceleration including ROCm and TensorRT.

Good fit when

When you need to serve embeddings and various models with high throughput and low latency.
If your deployment leverages GPUs for accelerated inference, especially in environments configured with ROCm or NVIDIA TensorRT.

Avoid when

Avoid using Infinity if your setup does not require GPU acceleration since its specialized Docker images may introduce unnecessary complexity.
Do not use Infinity if you are working with models that are not supported by it (such as specific NLP models outside of embeddings and reranking).

Observed Jul 12, 2026 · Source: enrich:decision_facts

infinity alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Slowing (136d since push); As of 2w
Provenance: Not a fork · Personal account; As of 2w
Security (OSV): No lockfile; As of 1mo

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install infinity

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

llmfit

Hundreds of models & providers. One command to find what runs on your hardware.

32k★

open-llms

A list of open LLMs available for commercial use.

13k★

clip-as-service

-scalable embedding, reasoning, ranking for images and sentences with CLIP-

13k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

Infinity provides a CLI and Docker container methods to serve embeddings, reranking models, CLIP, CLAP, and ColPaLi with support for GPU acceleration including ROCm and TensorRT.

Capability facts

Languages: python
Source: github.language · Aug 7, 2026

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Aug 7, 2026)

### Launch the cli via pip install

Source link

README

Launch the cli via pip install

pip install infinity-emb[all]

After your pip install, with your venv active, you can run the CLI directly.

infinity_emb v2 --model-id BAAI/bge-small-en-v1.5

Check the v2 --help command to get a description for all parameters.

infinity_emb v2 --help

Launch the CLI using a pre-built docker container (recommended)

Instead of installing the CLI via pip, you may also use docker to run michaelf34/infinity. Make sure you mount your accelerator ( i.e. install nvidia-docker and activate with --gpus all).

port=7997
model1=michaelfeil/bge-small-en-v1.5
model2=mixedbread-ai/mxbai-rerank-xsmall-v1
volume=$PWD/data

docker run -it --gpus all \
 -v $volume:/app/.cache \
 -p $port:$port \
 michaelf34/infinity:latest \
 v2 \
 --model-id $model1 \
 --model-id $model2 \
 --port $port

The cache path inside the docker container is set by the environment variable HF_HOME.

Specialized docker images

Docker container for CPU

Use the `latest-cpu` image or `x.x.x-cpu` for slimer image. Run like any other cpu-only docker image. Optimum/Onnx is often the prefered engine.

docker run -it \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest-cpu \
v2 \
--engine optimum \
--model-id $model1 \
--model-id $model2 \
--port $port

Docker Container for ROCm (MI200 Series and MI300 Series)

Use the `latest-rocm` image or `x.x.x-rocm` for rocm compatible inference. **This image is currently not build via CI/CD (to large), consider pinning to exact version.** Make sure you have ROCm is correctly installed and ready to use with Docker.

Visit Docs for more info.

Docker Container for Onnx-GPU, Cuda Extensions, TensorRT

Use the `latest-trt-onnx` image or `x.x.x-trt-onnx` for nvidia compatible inference. **This image is currently not build via CI/CD (to large), consider pinning to exact version.**

This image has support for:

ONNX-Cuda "CudaExecutionProvider"
ONNX-TensorRT "TensorRTExecutionProvider" (may not always work due to version mismatch with ORT)
CudaExtensions and packages, e.g. Tri-Dao's pip install flash-attn package when using Pytorch.
nvcc compiler support

docker run -it \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest-trt-onnx \
v2 \
--engine optimum \
--device cuda \
--model-id $model1 \
--port $port

Using local models with Docker container

In order to deploy a local model with a docker container, you need to mount the model inside the container and specify the path in the container to the launch command.

Example:

git lfs install 
cd /tmp
mkdir models && cd models && git clone https://huggingface.co/BAAI/bge-small-en-v1.5
docker run -it   -v /tmp/models:/models  -p 8081:8081  michaelf34/infinity:latest v2  --model-id "/models/bge-small-en-v1.5" --port 8081

Advanced CLI usage

Launching multiple models at once

Since infinity_emb>=0.0.34, you can use cli v2 method to launch multiple models at the same time. Checkout infinity_emb v2 --help for all args and validation.

Multiple Model CLI Playbook:

1. cli options can be repeated e.g. v2 --model-id model/id1 --model-id model/id2 --batch-size 8 --batch-size 4. This will create two models model/id1 and model/id2
1. or adapt the defaults by setting ENV Variables separated by ;: INFINITY_MODEL_ID="model/id1;model/id2;" && INFINITY_BATCH_SIZE="8;4;"
1. single items are broadcasted to --model-id length, v2 --model-id model/id1 --model-id/id2 --batch-size 8 making both models have batch-size 8.
1. Everythin

For agents

This page has a .md twin and JSON over the API.

/tools/michaelfeil-infinity.md GET /api/graphcanon/tools/michaelfeil-infinity GET /api/graphcanon/graph?tool=michaelfeil-infinity