text-generation-inference
Large Language Model Text Generation Inference
GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026
66views this month
Decision brief
text-generation-inference is a tool for serving large language models for text generation, supporting a wide range of hardware accelerators including Nvidia, AMD, Intel GPU, Gaudi, and Google TPU.
Good fit when
- When you need to serve large language models for text generation with support for multiple hardware accelerators, including AMD GPUs with ROCm.
- If you require compatibility with the Open AI Chat Completion API through TGI's Messages API.
Avoid when
- If your deployment environment lacks the necessary hardware accelerators, as performance on CPUs is not optimized and may be subpar.
- When you need a tool that does not require specific hardware configurations or Docker setup, as text-generation-inference is designed for environments with GPU support.
Observed Sep 18, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Archived (180d since push)
- As of Sep 18, 2026
- Provenance
- Not a fork · Organization account
- As of Sep 18, 2026
- Security (OSV)
- No lockfile
- As of Sep 18, 2026
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Backing
Company context for Hugging Face. Display-only - separate from trust and ranking.
- Company
- Hugging Face·GitHub org profile·Jul 11, 2026
- Employees
- 160·Wikidata (P1128 employees)·Jul 11, 2026
- Funding
- $235,000,000 (2023-08)·GraphCanon curated seed (public press)·Jul 11, 2026
- Commercial model
- OSS + managed cloud·GraphCanon curated seed·Jul 11, 2026
Install
pip install text-generation-inferenceHow it fits your stack(12)
Typed graph edges - alternatives, integrations, successors, and dependencies. Ranked by relationship type, not raw GitHub stars.
Integrates
Depends on
Related
Relationship graph
Optional deeper exploration of typed edges and category neighbours.
Similar tools
Same-category neighbours not already linked as typed edges.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
A tool for serving large language models for text generation, supporting various hardware accelerators including Nvidia, AMD, Intel GPU, Gaudi, and Google TPU.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Sep 18, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Sep 18, 2026
- Languages
- python
Source: github.language · Sep 18, 2026
Categories
Graph entities
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Sep 18, 2026)
Then [install Rust](https://rustup.rs/) and create a Python virtual environment with at leastSource link
Tags
README
Hardware support Nvidia AMD ( rocm) Inferentia Intel GPU Gaudi Google TPU Docker For a detailed starting guide, please see the Quick Tour. The easiest way of getting started is using the official Docker container: And then you can make requests like You can also use TGI's Message...
For agents
This page has a .md twin and JSON over the API.