{"data":{"slug":"huggingface-text-generation-inference","name":"text-generation-inference","tagline":"Large Language Model Text Generation Inference","github_url":"https://github.com/huggingface/text-generation-inference","owner":"huggingface","repo":"text-generation-inference","owner_avatar_url":"https://avatars.githubusercontent.com/u/25720743?v=4","primary_language":"Python","stars":10886,"forks":1287,"topics":["bloom","deep-learning","falcon","gpt","inference","nlp","pytorch","starcoder","transformer"],"archived":true,"github_pushed_at":"2026-03-21T11:34:22+00:00","maintenance_label":"Archived","stars_delta_30d":-2,"url":"https://www.graphcanon.com/tools/huggingface-text-generation-inference","markdown_url":"https://www.graphcanon.com/tools/huggingface-text-generation-inference.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/huggingface-text-generation-inference","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=huggingface-text-generation-inference","description":"Large Language Model Text Generation Inference","homepage_url":"http://hf.co/docs/text-generation-inference","license":"Apache-2.0","open_issues":324,"watchers":103,"ai_summary":"A tool for serving large language models for text generation, supporting various hardware accelerators including Nvidia, AMD, Intel GPU, Gaudi, and Google TPU.","readme_excerpt":"### Hardware support\n\n- [Nvidia](https://github.com/huggingface/text-generation-inference/pkgs/container/text-generation-inference)\n- [AMD](https://github.com/huggingface/text-generation-inference/pkgs/container/text-generation-inference) (-rocm)\n- [Inferentia](https://github.com/huggingface/optimum-neuron/tree/main/text-generation-inference)\n- [Intel GPU](https://github.com/huggingface/text-generation-inference/pull/1475)\n- [Gaudi](https://github.com/huggingface/tgi-gaudi)\n- [Google TPU](https://huggingface.co/docs/optimum-tpu/howto/serving)\n\n---\n\n### Docker\n\nFor a detailed starting guide, please see the [Quick Tour](https://huggingface.co/docs/text-generation-inference/quicktour). The easiest way of getting started is using the official Docker container:\n\n```shell\nmodel=HuggingFaceH4/zephyr-7b-beta\n\n---\n\n# share a volume with the Docker container to avoid downloading weights every run\nvolume=$PWD/data\n\ndocker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data \\\n    ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id $model\n```\n\nAnd then you can make requests like\n\n```bash\ncurl 127.0.0.1:8080/generate_stream \\\n    -X POST \\\n    -d '{\"inputs\":\"What is Deep Learning?\",\"parameters\":{\"max_new_tokens\":20}}' \\\n    -H 'Content-Type: application/json'\n```\n\nYou can also use [TGI's Messages API](https://huggingface.co/docs/text-generation-inference/en/messages_api) to obtain Open AI Chat Completion API compatible responses.\n\n```bash\ncurl localhost:8080/v1/chat/completions \\\n    -X POST \\\n    -d '{\n  \"model\": \"tgi\",\n  \"messages\": [\n    {\n      \"role\": \"system\",\n      \"content\": \"You are a helpful assistant.\"\n    },\n    {\n      \"role\": \"user\",\n      \"content\": \"What is deep learning?\"\n    }\n  ],\n  \"stream\": true,\n  \"max_tokens\": 20\n}' \\\n    -H 'Content-Type: application/json'\n```\n\n**Note:** To use NVIDIA GPUs, you need to install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). We also recommend using NVIDIA drivers with CUDA version 12.2 or higher. For running the Docker container on a machine with no GPUs or CUDA support, it is enough to remove the `--gpus all` flag and add `--disable-custom-kernels`, please note CPU is not the intended platform for this project, so performance might be subpar.\n\n**Note:** TGI supports AMD Instinct MI210 and MI250 GPUs. Details can be found in the [Supported Hardware documentation](https://huggingface.co/docs/text-generation-inference/installation_amd#using-tgi-with-amd-gpus). To use AMD GPUs, please use `docker run --device /dev/kfd --device /dev/dri --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:3.3.5-rocm --model-id $model` instead of the command above.\n\nTo see all options to serve your models (in the [code](https://github.com/huggingface/text-generation-inference/blob/main/launcher/src/main.rs) or in the cli):\n```\ntext-generation-launcher --help\n```\n\n---\n\n### Local install\n\nYou can also opt to install `text-generation-inference` locally.\n\nFirst clone the repository and change directory into it:\n\n```shell\ngit clone https://github.com/huggingface/text-generation-inference\ncd text-generation-inference\n```\n\nThen [install Rust](https://rustup.rs/) and create a Python virtual environment with at least\nPython 3.9, e.g. using `conda` or `python venv`:\n\n```shell\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n\n#using conda\nconda create -n text-generation-inference python=3.11\nconda activate text-generation-inference\n\n#using python venv\npython3 -m venv .venv\nsource .venv/bin/activate\n```\n\nYou may also need to install Protoc.\n\nOn Linux:\n\n```shell\nPROTOC_ZIP=protoc-21.12-linux-x86_64.zip\ncurl -OL https://github.com/protocolbuffers/protobuf/releases/download/v21.12/$PROTOC_ZIP\nsudo unzip -o $PROTOC_ZIP -d /usr/local bin/protoc\nsudo unzip -o $PROTOC_ZIP -d /usr/local 'include/*'\nrm -f $PROTOC_ZIP\n```\n\nOn MacOS, using Homebrew:\n\n```shell\nbrew install protobuf\n```\n\nThen run:","github_created_at":"2022-10-08T10:26:28+00:00","created_at":"2026-07-07T22:37:33.644122+00:00","updated_at":"2026-09-20T04:28:42.774971+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"bloom","name":"bloom"},{"slug":"deep-learning","name":"deep-learning"},{"slug":"falcon","name":"falcon"},{"slug":"gpt","name":"gpt"},{"slug":"inference","name":"inference"},{"slug":"nlp","name":"nlp"},{"slug":"pytorch","name":"pytorch"},{"slug":"starcoder","name":"starcoder"}],"trust":{"provenance":{"is_fork":false,"github_id":547806116,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:09:29.848Z","maintenance":{"label":"Archived","score":8,"methodology":"github_public_v1","releases_90d":0,"days_since_push":180,"last_release_at":"2025-12-19T14:35:25Z","stars_delta_30d":-2,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:09:32.353Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:09:28.706Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-09-18T08:09:28.706Z","managed_saas":false},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-09-18T08:09:28.706Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-09-18T08:09:28.706Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:09:28.706Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need to serve large language models for text generation with support for multiple hardware accelerators, including AMD GPUs with ROCm.","If you require compatibility with the Open AI Chat Completion API through TGI's Messages API.","When you want to use Docker containers for easy deployment and hardware-specific configurations, such as for NVIDIA GPUs with the NVIDIA Container Toolkit."],"when_not_to_use":["If your deployment environment lacks the necessary hardware accelerators, as performance on CPUs is not optimized and may be subpar.","When you need a tool that does not require specific hardware configurations or Docker setup, as text-generation-inference is designed for environments with GPU support."],"source":"enrich:decision_facts","observed_at":"2026-09-18T12:13:22.297Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"text-generation-inference is a tool for serving large language models for text generation, supporting a wide range of hardware accelerators including Nvidia, AMD, Intel GPU, Gaudi, and Google TPU."}]}}