{"data":{"slug":"huggingface-text-embeddings-inference","name":"text-embeddings-inference","tagline":"Blazing fast inference solution for text embeddings models","github_url":"https://github.com/huggingface/text-embeddings-inference","owner":"huggingface","repo":"text-embeddings-inference","owner_avatar_url":"https://avatars.githubusercontent.com/u/25720743?v=4","primary_language":"Rust","stars":5054,"forks":427,"topics":["ai","embeddings","huggingface","llm","ml"],"archived":false,"github_pushed_at":"2026-09-17T13:13:59+00:00","maintenance_label":"Very active","stars_delta_30d":72,"url":"https://www.graphcanon.com/tools/huggingface-text-embeddings-inference","markdown_url":"https://www.graphcanon.com/tools/huggingface-text-embeddings-inference.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/huggingface-text-embeddings-inference","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=huggingface-text-embeddings-inference","description":"A blazing fast inference solution for text embeddings models","homepage_url":"https://huggingface.co/docs/text-embeddings-inference/quick_tour","license":"Apache-2.0","open_issues":205,"watchers":43,"ai_summary":"A Rust-based tool for fast inference of text embeddings models, deployable via Docker with GPU support.","readme_excerpt":"### Docker\n\n```shell\nmodel=Qwen/Qwen3-Embedding-0.6B\nvolume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run\n\ndocker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model\n```\n\nAnd then you can make requests like\n\n```bash\ncurl 127.0.0.1:8080/embed \\\n    -X POST \\\n    -d '{\"inputs\":\"What is Deep Learning?\"}' \\\n    -H 'Content-Type: application/json'\n```\n\n**Note:** To use GPUs, you need to install\nthe [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html).\nNVIDIA drivers on your machine need to be compatible with CUDA version 12.2 or higher.\n\nTo see all options to serve your models:\n\n```console\n$ text-embeddings-router --help\nText Embedding Webserver\n\nUsage: text-embeddings-router [OPTIONS] --model-id <MODEL_ID>\n\nOptions:\n      --model-id <MODEL_ID>\n          The Hugging Face model ID, can be any model listed on <https://huggingface.co/models> with the `text-embeddings-inference` tag (meaning it's compatible with Text Embeddings Inference).\n\n          Alternatively, the specified ID can also be a path to a local directory containing the necessary model files saved by the `save_pretrained(...)` methods of either Transformers or Sentence Transformers.\n\n          [env: MODEL_ID=]\n\n      --revision <REVISION>\n          The actual revision of the model if you're referring to a model on the hub. You can use a specific commit id or a branch like `refs/pr/2`\n\n          [env: REVISION=]\n\n      --tokenization-workers <TOKENIZATION_WORKERS>\n          Optionally control the number of tokenizer workers used for payload tokenization, validation and truncation. Default to the number of CPU cores on the machine\n\n          [env: TOKENIZATION_WORKERS=]\n\n      --dtype <DTYPE>\n          The dtype to be forced upon the model\n\n          [env: DTYPE=]\n          [possible values: float16, float32]\n\n      --served-model-name <SERVED_MODEL_NAME>\n          The name of the model that is being served. If not specified, defaults to `--model-id`. It is only used for the OpenAI-compatible endpoints via HTTP\n\n          [env: SERVED_MODEL_NAME=]\n\n      --pooling <POOLING>\n          Optionally control the pooling method for embedding models.\n\n          If `pooling` is not set, the pooling configuration will be parsed from the model `1_Pooling/config.json` configuration.\n\n          If `pooling` is set, it will override the model pooling configuration\n\n          [env: POOLING=]\n\n          Possible values:\n          - cls:        Select the CLS token as embedding\n          - mean:       Apply Mean pooling to the model embeddings\n          - splade:     Apply SPLADE (Sparse Lexical and Expansion) to the model embeddings. This option is only available if the loaded model is a `ForMaskedLM` Transformer model\n          - last-token: Select the last token as embedding\n\n      --max-concurrent-requests <MAX_CONCURRENT_REQUESTS>\n          The maximum amount of concurrent requests for this particular deployment. Having a low limit will refuse clients requests instead of having them wait for too long and is usually good to handle backpressure correctly\n\n          [env: MAX_CONCURRENT_REQUESTS=]\n          [default: 512]\n\n      --max-batch-tokens <MAX_BATCH_TOKENS>\n          **IMPORTANT** This is one critical control to allow maximum usage of the available hardware.\n\n          This represents the total amount of potential tokens within a batch.\n\n          For `max_batch_tokens=1000`, you could fit `10` queries of `total_tokens=100` or a single query of `1000` tokens.\n\n          Overall this number should be the largest possible until the model is compute bound. Since the actual memory overhead depends on the model implementation, text-embeddings-inference cannot infer this number automatically.\n\n          [env: MAX_BATCH_TOKENS=]\n          [default: 16384]\n\n      --max-batch-requests <MAX_BATCH_REQUESTS>","github_created_at":"2023-10-13T13:36:51+00:00","created_at":"2026-07-11T10:37:43.791731+00:00","updated_at":"2026-09-20T04:29:49.35359+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"ai","name":"ai"},{"slug":"docker","name":"docker"},{"slug":"embeddings","name":"embeddings"},{"slug":"gpu","name":"gpu"},{"slug":"huggingface","name":"huggingface"},{"slug":"llm","name":"llm"},{"slug":"ml","name":"ml"},{"slug":"rust","name":"rust"}],"trust":{"provenance":{"is_fork":false,"github_id":704543233,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:18:50.625Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":1,"days_since_push":0,"last_release_at":"2026-09-15T09:28:03Z","stars_delta_30d":72,"open_issues_delta_30d":1},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:18:52.520Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:18:49.513Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-09-18T08:18:49.513Z","managed_saas":false},"languages":{"value":["rust"],"source":"github.language","observed_at":"2026-09-18T08:18:49.513Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-09-18T08:18:49.513Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:18:49.513Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium","summary":"The tool is free to use under the Apache-2.0 license, but users may incur costs based on their infrastructure and usage."},"requirements":{"notes":["NVIDIA Container Toolkit must be installed for GPU support.","Compatible NVIDIA drivers are required for CUDA version 12.2 or higher."],"min_ram_gb":4,"requires_docker":true},"constraints":{"min_ram_gb":4,"pricing_model":"freemium","requires_docker":true},"when_to_use":["When you need blazing fast inference for text embeddings models, as this tool is optimized for speed.","If you are working with Hugging Face models and want to leverage GPU acceleration for faster inference.","When you require a solution that can handle a high number of concurrent requests efficiently, thanks to its configurable max-concurrent-requests parameter."],"when_not_to_use":["If your environment does not support CUDA version 12.2 or higher, as the tool requires compatible NVIDIA drivers for GPU support.","When you do not have access to a Docker environment or do not wish to use Docker for deployment.","If you need a tool that supports more than just text embeddings models, as this tool is specifically designed for text embeddings inference."],"source":"enrich:decision_facts","observed_at":"2026-09-18T10:16:44.771Z"},"constraint_facets":{"min_ram_gb":4,"pricing_model":"freemium","requires_docker":true},"decision_summary":[{"label":"Pricing","value":"freemium - The tool is free to use under the Apache-2.0 license, but users may incur costs based on their infrastructure and usage."},{"label":"Requirements","value":"Min 4 GB RAM; Requires Docker; NVIDIA Container Toolkit must be installed for GPU support.; Compatible NVIDIA drivers are required for CUDA version 12.2 or higher."},{"label":"Adopt for","value":"text-embeddings-inference is a Rust-based tool for fast inference of text embeddings models, deployable via Docker with GPU support."}]}}