{"data":{"slug":"huggingface-text-generation-inference","name":"text-generation-inference","tagline":"Large Language Model Text Generation Inference","github_url":"https://github.com/huggingface/text-generation-inference","owner":"huggingface","repo":"text-generation-inference","owner_avatar_url":"https://avatars.githubusercontent.com/u/25720743?v=4","primary_language":"Python","stars":10867,"forks":1270,"topics":["bloom","deep-learning","falcon","gpt","inference","nlp","pytorch","starcoder","transformer"],"archived":true,"github_pushed_at":"2026-03-21T11:34:22+00:00","url":"https://www.graphcanon.com/tools/huggingface-text-generation-inference","markdown_url":"https://www.graphcanon.com/tools/huggingface-text-generation-inference.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/huggingface-text-generation-inference","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=huggingface-text-generation-inference","description":"Large Language Model Text Generation Inference","homepage_url":"http://hf.co/docs/text-generation-inference","license":"Apache-2.0","open_issues":324,"watchers":100,"ai_summary":"A Rust, Python and gRPC server for text generation inference used in production at Hugging Face. Now in maintenance mode.","readme_excerpt":"> [!CAUTION]\n> text-generation-inference is now in maintenance mode. Going forward, we will accept pull requests for minor bug fixes, documentation improvements and lightweight maintenance tasks.\n>\n> TGI has initiated the movement for optimized inference engines to rely on a `transformers` model architectures. This approach is now adopted by downstream inference engines, which we contribute to and recommend using going forward: [vllm](https://github.com/vllm-project/vllm), [SGLang](https://github.com/sgl-project/sglang), as well as local engines with inter-compatibility such as llama.cpp or MLX.\n\n
\n\n\n \"Making\n\n\n# Text Generation Inference\n\n\n \"GitHub\n\n\n \"Swagger\n\n\nA Rust, Python and gRPC server for text generation inference. Used in production at [Hugging Face](https://huggingface.co)\nto power Hugging Chat, the Inference API and Inference Endpoints.\n\n
\n\n## Table of contents\n\n - [Get Started](#get-started)\n - [Docker](#docker)\n - [API documentation](#api-documentation)\n - [Using a private or gated model](#using-a-private-or-gated-model)\n - [A note on Shared Memory (shm)](#a-note-on-shared-memory-shm)\n - [Distributed Tracing](#distributed-tracing)\n - [Architecture](#architecture)\n - [Local install](#local-install)\n - [Local install (Nix)](#local-install-nix)\n - [Optimized architectures](#optimized-architectures)\n - [Run locally](#run-locally)\n - [Run](#run)\n - [Quantization](#quantization)\n - [Develop](#develop)\n - [Testing](#testing)\n\nText Generation Inference (TGI) is a toolkit for deploying and serving Large Language Models (LLMs). TGI enables high-performance text generation for the most popular open-source LLMs, including Llama, Falcon, StarCoder, BLOOM, GPT-NeoX, and [more](https://huggingface.co/docs/text-generation-inference/supported_models). TGI implements many features, such as:\n\n- Simple launcher to serve most popular LLMs\n- Production ready (distributed tracing with Open Telemetry, Prometheus metrics)\n- Tensor Parallelism for faster inference on multiple GPUs\n- Token streaming using Server-Sent Events (SSE)\n- Continuous batching of incoming requests for increased total throughput\n- [Messages API](https://huggingface.co/docs/text-generation-inference/en/messages_api) compatible with Open AI Chat Completion API\n- Optimized transformers code for inference using [Flash Attention](https://github.com/HazyResearch/flash-attention) and [Paged Attention](https://github.com/vllm-project/vllm) on the most popular architectures\n- Quantization with :\n - [bitsandbytes](https://github.com/TimDettmers/bitsandbytes)\n - [GPT-Q](https://arxiv.org/abs/2210.17323)\n - [EETQ](https://github.com/NetEase-FuXi/EETQ)\n - [AWQ](https://github.com/casper-hansen/AutoAWQ)\n - [Marlin](https://github.com/IST-DASLab/marlin)\n - [fp8](https://developer.nvidia.com/blog/nvidia-arm-and-intel-publish-fp8-specification-for-standardization-as-an-interchange-format-for-ai/)\n- [Safetensors](https://github.com/huggingface/safetensors) weight loading\n- Watermarking with [A Watermark for Large Language Models](https://arxiv.org/abs/2301.10226)\n- Logits warper (temperature scaling, top-p, top-k, repetition penalty, more details see [transformers.LogitsProcessor](https://huggingface.co/docs/transformers/internal/generation_utils#transformers.LogitsProcessor))\n- Stop sequences\n- Log probabilities\n- [Speculation](https://huggingface.co/docs/text-generation-inference/conceptual/spe","github_created_at":"2022-10-08T10:26:28+00:00","created_at":"2026-07-07T22:37:33.644122+00:00","updated_at":"2026-07-07T22:37:36.644492+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"developer-tools","name":"Developer Tools","url":"https://www.graphcanon.com/categories/developer-tools","markdown_url":"https://www.graphcanon.com/categories/developer-tools.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/developer-tools"}],"tags":[{"slug":"deep-learning","name":"deep-learning"},{"slug":"nlp","name":"nlp"},{"slug":"falcon","name":"falcon"},{"slug":"starcoder","name":"starcoder"},{"slug":"gpt","name":"gpt"},{"slug":"pytorch","name":"pytorch"},{"slug":"bloom","name":"bloom"},{"slug":"inference","name":"inference"}]}}