{"data":{"slug":"michaelfeil-infinity","name":"infinity","tagline":"High-throughput, low-latency serving engine for text-embeddings, reranking models, CLIP, CLAP, and COLPAli","github_url":"https://github.com/michaelfeil/infinity","owner":"michaelfeil","repo":"infinity","owner_avatar_url":"https://avatars.githubusercontent.com/u/63565275?v=4","primary_language":"Python","stars":2942,"forks":207,"topics":["bert-embeddings","llm","text-embeddings"],"archived":false,"github_pushed_at":"2026-03-24T03:59:47+00:00","maintenance_label":"Slowing","stars_delta_30d":35,"url":"https://www.graphcanon.com/tools/michaelfeil-infinity","markdown_url":"https://www.graphcanon.com/tools/michaelfeil-infinity.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/michaelfeil-infinity","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=michaelfeil-infinity","description":"Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali","homepage_url":"https://michaelfeil.github.io/infinity/","license":"MIT","open_issues":136,"watchers":22,"ai_summary":"Infinity provides a CLI and Docker containers for serving text-embeddings and reranking models with support for various hardware accelerators.","readme_excerpt":"### Launch the cli via pip install\n```bash\npip install infinity-emb[all]\n```\nAfter your pip install, with your venv active, you can run the CLI directly.\n\n```bash\ninfinity_emb v2 --model-id BAAI/bge-small-en-v1.5\n```\nCheck the `v2 --help` command to get a description for all parameters.\n```bash\ninfinity_emb v2 --help\n```\n\n---\n\n### Launch the CLI using a pre-built docker container (recommended)\nInstead of installing the CLI via pip, you may also use docker to run `michaelf34/infinity`. \nMake sure you mount your accelerator ( i.e. install `nvidia-docker` and activate with `--gpus all`). \n\n```bash\nport=7997\nmodel1=michaelfeil/bge-small-en-v1.5\nmodel2=mixedbread-ai/mxbai-rerank-xsmall-v1\nvolume=$PWD/data\n\ndocker run -it --gpus all \\\n -v $volume:/app/.cache \\\n -p $port:$port \\\n michaelf34/infinity:latest \\\n v2 \\\n --model-id $model1 \\\n --model-id $model2 \\\n --port $port\n```\nThe cache path inside the docker container is set by the environment variable `HF_HOME`.\n\n#### Specialized docker images\n\n Docker container for CPU
\n Use the `latest-cpu` image or `x.x.x-cpu` for slimer image. \n Run like any other cpu-only docker image. \n Optimum/Onnx is often the prefered engine. \n\n ```\n docker run -it \\\n -v $volume:/app/.cache \\\n -p $port:$port \\\n michaelf34/infinity:latest-cpu \\\n v2 \\\n --engine optimum \\\n --model-id $model1 \\\n --model-id $model2 \\\n --port $port\n ```\n \n\n\n Docker Container for ROCm (MI200 Series and MI300 Series)
\n Use the `latest-rocm` image or `x.x.x-rocm` for rocm compatible inference.\n **This image is currently not build via CI/CD (to large), consider pinning to exact version.**\n Make sure you have ROCm is correctly installed and ready to use with Docker.\n\n Visit [Docs](https://michaelfeil.github.io/infinity) for more info.\n \n \n\n Docker Container for Onnx-GPU, Cuda Extensions, TensorRT
\n Use the `latest-trt-onnx` image or `x.x.x-trt-onnx` for nvidia compatible inference.\n **This image is currently not build via CI/CD (to large), consider pinning to exact version.**\n\n This image has support for:\n - ONNX-Cuda \"CudaExecutionProvider\" \n - ONNX-TensorRT \"TensorRTExecutionProvider\" (may not always work due to version mismatch with ORT)\n - CudaExtensions and packages, e.g. Tri-Dao's `pip install flash-attn` package when using Pytorch.\n - nvcc compiler support\n \n ```\n docker run -it \\\n -v $volume:/app/.cache \\\n -p $port:$port \\\n michaelf34/infinity:latest-trt-onnx \\\n v2 \\\n --engine optimum \\\n --device cuda \\\n --model-id $model1 \\\n --port $port\n ```\n \n\n#### Using local models with Docker container\n\nIn order to deploy a local model with a docker container, you need to mount the model inside the container and specify the path in the container to the launch command.\n\nExample:\n```bash\ngit lfs install \ncd /tmp\nmkdir models && cd models && git clone https://huggingface.co/BAAI/bge-small-en-v1.5\ndocker run -it -v /tmp/models:/models -p 8081:8081 michaelf34/infinity:latest v2 --model-id \"/models/bge-small-en-v1.5\" --port 8081\n```\n\n#### Advanced CLI usage\n\n\n Launching multiple models at once
\n \n Since `infinity_emb>=0.0.34`, you can use cli `v2` method to launch multiple models at the same time.\n Checkout `infinity_emb v2 --help` for all args and validation.\n\n Multiple Model CLI Playbook: \n - 1. cli options can be repeated e.g. `v2 --model-id model/id1 --model-id model/id2 --batch-size 8 --batch-size 4`. This will create two models `model/id1` and `model/id2`\n - 2. or adapt the defaults by setting ENV Variables separated by `;`: `INFINITY_MODEL_ID=\"model/id1;model/id2;\" && INFINITY_BATCH_SIZE=\"8;4;\"`\n - 3. single items are broadcasted to `--model-id` length, `v2 --model-id model/id1 --model-id/id2 --batch-size 8` making both models have batch-size 8.\n - 4. Everythin","github_created_at":"2023-10-11T17:53:38+00:00","created_at":"2026-07-11T10:37:48.740545+00:00","updated_at":"2026-09-20T04:29:43.096752+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"bert-embeddings","name":"bert-embeddings"},{"slug":"cpu","name":"cpu"},{"slug":"docker","name":"docker"},{"slug":"gpu","name":"gpu"},{"slug":"llm","name":"llm"},{"slug":"onnx","name":"onnx"},{"slug":"rocm","name":"rocm"},{"slug":"text-embeddings","name":"text-embeddings"}],"trust":{"provenance":{"is_fork":false,"github_id":703686617,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:17:59.960Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":178,"last_release_at":"2025-08-22T23:15:05Z","stars_delta_30d":35,"open_issues_delta_30d":6},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:18:02.330Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:17:58.872Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-09-18T08:17:58.872Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-09-18T08:17:58.872Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["Use Infinity when you need to serve text-embeddings and reranking models with high throughput and low latency, especially when you have access to hardware accelerators like GPUs or ROCm.","Consider Infinity if you require support for multiple models simultaneously, as it allows launching multiple models at once with the CLI.","Use Infinity when you prefer to deploy models via Docker containers, as it provides specialized images for different hardware configurations, including CPU, ROCm, and ONNX-GPU."],"when_not_to_use":["Avoid using Infinity if you do not have access to hardware accelerators and prefer a lightweight solution, as the specialized Docker images for ROCm and ONNX-GPU are not built via CI/CD and may not be","Do not use Infinity if you need a serving engine that is frequently updated via CI/CD, as some specialized Docker images are not automatically built and may require manual version pinning."],"source":"enrich:decision_facts","observed_at":"2026-09-18T09:59:28.939Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, CLIP, CLAP, and COLPAli, offering support for various hardware accelerators via CLI and Docker containers."}]}}