{"data":{"slug":"huggingface-text-generation-inference","name":"text-generation-inference","tagline":"Large Language Model Text Generation Inference","github_url":"https://github.com/huggingface/text-generation-inference","owner":"huggingface","repo":"text-generation-inference","owner_avatar_url":"https://avatars.githubusercontent.com/u/25720743?v=4","primary_language":"Python","stars":10888,"forks":1274,"topics":["bloom","deep-learning","falcon","gpt","inference","nlp","pytorch","starcoder","transformer"],"archived":true,"github_pushed_at":"2026-03-21T11:34:22+00:00","maintenance_label":"Archived","url":"https://www.graphcanon.com/tools/huggingface-text-generation-inference","markdown_url":"https://www.graphcanon.com/tools/huggingface-text-generation-inference.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/huggingface-text-generation-inference","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=huggingface-text-generation-inference","description":"Large Language Model Text Generation Inference","homepage_url":"http://hf.co/docs/text-generation-inference","license":"Apache-2.0","open_issues":324,"watchers":100,"ai_summary":"Provides infrastructure for large language model inference, supporting hardware like NVIDIA GPUs, AMD ROCm, Gaudi, Intel GPU, and Google TPU.","readme_excerpt":"### Hardware support\n\n- [Nvidia](https://github.com/huggingface/text-generation-inference/pkgs/container/text-generation-inference)\n- [AMD](https://github.com/huggingface/text-generation-inference/pkgs/container/text-generation-inference) (-rocm)\n- [Inferentia](https://github.com/huggingface/optimum-neuron/tree/main/text-generation-inference)\n- [Intel GPU](https://github.com/huggingface/text-generation-inference/pull/1475)\n- [Gaudi](https://github.com/huggingface/tgi-gaudi)\n- [Google TPU](https://huggingface.co/docs/optimum-tpu/howto/serving)\n\n---\n\n### Docker\n\nFor a detailed starting guide, please see the [Quick Tour](https://huggingface.co/docs/text-generation-inference/quicktour). The easiest way of getting started is using the official Docker container:\n\n```shell\nmodel=HuggingFaceH4/zephyr-7b-beta\n\n---\n\n# share a volume with the Docker container to avoid downloading weights every run\nvolume=$PWD/data\n\ndocker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data \\\n    ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id $model\n```\n\nAnd then you can make requests like\n\n```bash\ncurl 127.0.0.1:8080/generate_stream \\\n    -X POST \\\n    -d '{\"inputs\":\"What is Deep Learning?\",\"parameters\":{\"max_new_tokens\":20}}' \\\n    -H 'Content-Type: application/json'\n```\n\nYou can also use [TGI's Messages API](https://huggingface.co/docs/text-generation-inference/en/messages_api) to obtain Open AI Chat Completion API compatible responses.\n\n```bash\ncurl localhost:8080/v1/chat/completions \\\n    -X POST \\\n    -d '{\n  \"model\": \"tgi\",\n  \"messages\": [\n    {\n      \"role\": \"system\",\n      \"content\": \"You are a helpful assistant.\"\n    },\n    {\n      \"role\": \"user\",\n      \"content\": \"What is deep learning?\"\n    }\n  ],\n  \"stream\": true,\n  \"max_tokens\": 20\n}' \\\n    -H 'Content-Type: application/json'\n```\n\n**Note:** To use NVIDIA GPUs, you need to install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). We also recommend using NVIDIA drivers with CUDA version 12.2 or higher. For running the Docker container on a machine with no GPUs or CUDA support, it is enough to remove the `--gpus all` flag and add `--disable-custom-kernels`, please note CPU is not the intended platform for this project, so performance might be subpar.\n\n**Note:** TGI supports AMD Instinct MI210 and MI250 GPUs. Details can be found in the [Supported Hardware documentation](https://huggingface.co/docs/text-generation-inference/installation_amd#using-tgi-with-amd-gpus). To use AMD GPUs, please use `docker run --device /dev/kfd --device /dev/dri --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:3.3.5-rocm --model-id $model` instead of the command above.\n\nTo see all options to serve your models (in the [code](https://github.com/huggingface/text-generation-inference/blob/main/launcher/src/main.rs) or in the cli):\n```\ntext-generation-launcher --help\n```\n\n---\n\n### Local install\n\nYou can also opt to install `text-generation-inference` locally.\n\nFirst clone the repository and change directory into it:\n\n```shell\ngit clone https://github.com/huggingface/text-generation-inference\ncd text-generation-inference\n```\n\nThen [install Rust](https://rustup.rs/) and create a Python virtual environment with at least\nPython 3.9, e.g. using `conda` or `python venv`:\n\n```shell\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n\n#using conda\nconda create -n text-generation-inference python=3.11\nconda activate text-generation-inference\n\n#using python venv\npython3 -m venv .venv\nsource .venv/bin/activate\n```\n\nYou may also need to install Protoc.\n\nOn Linux:\n\n```shell\nPROTOC_ZIP=protoc-21.12-linux-x86_64.zip\ncurl -OL https://github.com/protocolbuffers/protobuf/releases/download/v21.12/$PROTOC_ZIP\nsudo unzip -o $PROTOC_ZIP -d /usr/local bin/protoc\nsudo unzip -o $PROTOC_ZIP -d /usr/local 'include/*'\nrm -f $PROTOC_ZIP\n```\n\nOn MacOS, using Homebrew:\n\n```shell\nbrew install protobuf\n```\n\nThen run:","github_created_at":"2022-10-08T10:26:28+00:00","created_at":"2026-07-07T22:37:33.644122+00:00","updated_at":"2026-08-06T06:02:15.280017+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"bloom","name":"bloom"},{"slug":"deep-learning","name":"deep-learning"},{"slug":"falcon","name":"falcon"},{"slug":"gpt","name":"gpt"},{"slug":"inference","name":"inference"},{"slug":"nlp","name":"nlp"},{"slug":"pytorch","name":"pytorch"},{"slug":"starcoder","name":"starcoder"}],"trust":{"provenance":{"is_fork":false,"github_id":547806116,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-06T06:02:14.533Z","maintenance":{"label":"Archived","score":8,"methodology":"github_public_v1","releases_90d":0,"days_since_push":137,"last_release_at":"2025-12-19T14:35:25Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:29:39.951Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-06T06:02:14.951Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-08-06T06:02:14.951Z","managed_saas":false},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-06T06:02:14.951Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-08-06T06:02:14.951Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-06T06:02:14.951Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium","summary":"Available under the Apache-2.0 license with a community-maintained open-source model."},"requirements":{"notes":["NVIDIA GPUs require NVIDIA Container Toolkit and CUDA drivers 12.2 or higher.","AMD ROCm support requires AMD Instinct MI210 or MI250 series with appropriate setup."],"min_ram_gb":4,"requires_docker":true},"constraints":{"min_ram_gb":4,"pricing_model":"freemium","requires_docker":true},"when_to_use":["When you need hardware-accelerated performance on a variety of GPUs including NVIDIA (with CUDA 12.2 or higher), AMD ROCm, Intel GPU, Gaudi, and Google TPU.","If your workflow includes models from Bloom, Falcon, StarCoder, or similar transformers-based architectures that benefit from high-performance inference engines.","For deep learning text generation tasks where you require low-latency response and efficient resource utilization."],"when_not_to_use":["When the target hardware lacks GPU support or does not match the supported platforms (e.g., non-NVIDIA GPUs without ROCm setup).","If you need high-performance on CPUs exclusively, as TGI is designed primarily for GPU acceleration and CPU performance might be subpar.","For model training tasks; TGI focuses specifically on inference rather than training large language models."],"source":"enrich:decision_facts","observed_at":"2026-07-11T11:05:45.170Z"},"constraint_facets":{"min_ram_gb":4,"pricing_model":"freemium","requires_docker":true},"decision_summary":[{"label":"Pricing","value":"freemium - Available under the Apache-2.0 license with a community-maintained open-source model."},{"label":"Requirements","value":"Min 4 GB RAM; Requires Docker; NVIDIA GPUs require NVIDIA Container Toolkit and CUDA drivers 12.2 or higher.; AMD ROCm support requires AMD Instinct MI210 or MI250 series with appropriate setup."},{"label":"Adopt for","value":"text-generation-inference"}]}}