{"data":{"slug":"michaelfeil-infinity","name":"infinity","tagline":"High-throughput, low-latency serving engine for text-embeddings and various models","github_url":"https://github.com/michaelfeil/infinity","owner":"michaelfeil","repo":"infinity","owner_avatar_url":"https://avatars.githubusercontent.com/u/63565275?v=4","primary_language":"Python","stars":2874,"forks":196,"topics":["bert-embeddings","llm","text-embeddings"],"archived":false,"github_pushed_at":"2026-03-24T03:59:47+00:00","maintenance_label":"Slowing","url":"https://www.graphcanon.com/tools/michaelfeil-infinity","markdown_url":"https://www.graphcanon.com/tools/michaelfeil-infinity.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/michaelfeil-infinity","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=michaelfeil-infinity","description":"Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali","homepage_url":"https://michaelfeil.github.io/infinity/","license":"MIT","open_issues":130,"watchers":22,"ai_summary":"Infinity provides a CLI and Docker container methods to serve embeddings, reranking models, CLIP, CLAP, and ColPaLi with support for GPU acceleration including ROCm and TensorRT.","readme_excerpt":"### Launch the cli via pip install\n```bash\npip install infinity-emb[all]\n```\nAfter your pip install, with your venv active, you can run the CLI directly.\n\n```bash\ninfinity_emb v2 --model-id BAAI/bge-small-en-v1.5\n```\nCheck the `v2 --help` command to get a description for all parameters.\n```bash\ninfinity_emb v2 --help\n```\n\n---\n\n### Launch the CLI using a pre-built docker container (recommended)\nInstead of installing the CLI via pip, you may also use docker to run `michaelf34/infinity`. \nMake sure you mount your accelerator ( i.e. install `nvidia-docker` and activate with `--gpus all`). \n\n```bash\nport=7997\nmodel1=michaelfeil/bge-small-en-v1.5\nmodel2=mixedbread-ai/mxbai-rerank-xsmall-v1\nvolume=$PWD/data\n\ndocker run -it --gpus all \\\n -v $volume:/app/.cache \\\n -p $port:$port \\\n michaelf34/infinity:latest \\\n v2 \\\n --model-id $model1 \\\n --model-id $model2 \\\n --port $port\n```\nThe cache path inside the docker container is set by the environment variable `HF_HOME`.\n\n#### Specialized docker images\n<details>\n  <summary>Docker container for CPU</summary>\n  Use the `latest-cpu` image or `x.x.x-cpu` for slimer image. \n  Run like any other cpu-only docker image. \n  Optimum/Onnx is often the prefered engine. \n\n  ```\n  docker run -it \\\n  -v $volume:/app/.cache \\\n  -p $port:$port \\\n  michaelf34/infinity:latest-cpu \\\n  v2 \\\n  --engine optimum \\\n  --model-id $model1 \\\n  --model-id $model2 \\\n  --port $port\n  ```\n</details>\n\n<details>\n  <summary>Docker Container for ROCm (MI200 Series and MI300 Series)</summary>\n  Use the `latest-rocm` image or `x.x.x-rocm` for rocm compatible inference.\n  **This image is currently not build via CI/CD (to large), consider pinning to exact version.**\n  Make sure you have ROCm is correctly installed and ready to use with Docker.\n\n  Visit [Docs](https://michaelfeil.github.io/infinity) for more info.\n</details>\n \n<details>\n  <summary>Docker Container for Onnx-GPU, Cuda Extensions, TensorRT</summary>\n  Use the `latest-trt-onnx` image or `x.x.x-trt-onnx` for nvidia compatible inference.\n  **This image is currently not build via CI/CD (to large), consider pinning to exact version.**\n\n  This image has support for:\n  - ONNX-Cuda \"CudaExecutionProvider\" \n  - ONNX-TensorRT \"TensorRTExecutionProvider\" (may not always work due to version mismatch with ORT)\n  - CudaExtensions and packages, e.g. Tri-Dao's `pip install flash-attn` package when using Pytorch.\n  - nvcc compiler support\n  \n  ```\n  docker run -it \\\n  -v $volume:/app/.cache \\\n  -p $port:$port \\\n  michaelf34/infinity:latest-trt-onnx \\\n  v2 \\\n  --engine optimum \\\n  --device cuda \\\n  --model-id $model1 \\\n  --port $port\n  ```\n</details>\n\n#### Using local models with Docker container\n\nIn order to deploy a local model with a docker container, you need to mount the model inside the container and specify the path in the container to the launch command.\n\nExample:\n```bash\ngit lfs install \ncd /tmp\nmkdir models && cd models && git clone https://huggingface.co/BAAI/bge-small-en-v1.5\ndocker run -it   -v /tmp/models:/models  -p 8081:8081  michaelf34/infinity:latest v2  --model-id \"/models/bge-small-en-v1.5\" --port 8081\n```\n\n#### Advanced CLI usage\n\n<details>\n  <summary>Launching multiple models at once</summary>\n  \n  Since `infinity_emb>=0.0.34`, you can use cli `v2` method to launch multiple models at the same time.\n  Checkout `infinity_emb v2 --help` for all args and validation.\n\n  Multiple Model CLI Playbook:                                                                                         \n   - 1. cli options can be repeated e.g. `v2 --model-id model/id1 --model-id model/id2 --batch-size 8 --batch-size 4`. This will create two models `model/id1` and `model/id2`\n   - 2. or adapt the defaults by setting ENV Variables separated by `;`: `INFINITY_MODEL_ID=\"model/id1;model/id2;\" && INFINITY_BATCH_SIZE=\"8;4;\"`\n   - 3. single items are broadcasted to `--model-id` length,  `v2 --model-id model/id1 --model-id/id2 --batch-size 8` making both models have batch-size 8.\n   - 4. Everythin","github_created_at":"2023-10-11T17:53:38+00:00","created_at":"2026-07-11T10:37:48.740545+00:00","updated_at":"2026-07-12T07:59:08.740683+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"clap","name":"clap"},{"slug":"clip","name":"clip"},{"slug":"colpali","name":"colpali"},{"slug":"docker-container","name":"docker-container"},{"slug":"gpu-acceleration","name":"gpu-acceleration"},{"slug":"llm","name":"llm"},{"slug":"text-embeddings","name":"text-embeddings"}],"trust":{"provenance":{"is_fork":false,"github_id":703686617,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:37:49.359Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":109,"last_release_at":"2025-08-22T23:15:05Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:37:50.663Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-12T07:58:44.716Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-12T07:58:44.716Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-12T07:58:44.716Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need to serve embeddings and various models with high throughput and low latency.","If your deployment leverages GPUs for accelerated inference, especially in environments configured with ROCm or NVIDIA TensorRT."],"when_not_to_use":["Avoid using Infinity if your setup does not require GPU acceleration since its specialized Docker images may introduce unnecessary complexity.","Do not use Infinity if you are working with models that are not supported by it (such as specific NLP models outside of embeddings and reranking)."],"source":"enrich:decision_facts","observed_at":"2026-07-12T07:59:08.528Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Infinity is a high-throughput, low-latency serving engine that supports text-embeddings, reranking models, CLIP, CLAP, and ColPaLi, with GPU acceleration including ROCm and TensorRT."}]}}