Home/Inference & Serving/TensorRT-LLM

TensorRT-LLM

NVIDIA/TensorRT-LLM

Python API for defining and optimizing Large Language Models for efficient inference on NVIDIA GPUs

GitHub Website

GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026

82views this month

15k stars2.8k forksLast push Sep 18, 2026 Python Other

Decision brief

TensorRT-LLM is a Python API for defining and optimizing Large Language Models for efficient inference on NVIDIA GPUs, offering components for creating Python and C++ runtimes for performant inference execution.

Good fit when

When you need to optimize Large Language Models for efficient inference on NVIDIA GPUs.
If you are working with NVIDIA GPUs and require a Python API for defining and optimizing LLMs.

Avoid when

If your hardware does not include NVIDIA GPUs, as TensorRT-LLM is specifically optimized for NVIDIA GPUs.
If you do not require a Python API for defining and optimizing LLMs, and prefer a different interface or tool.

Observed Sep 18, 2026 · Source: enrich:decision_facts

TensorRT-LLM alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Very active (0d since push); As of Sep 18, 2026
Provenance: Not a fork · Organization account; As of Sep 18, 2026
Security (OSV): 11 low (11 low); As of Sep 18, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Backing

Company context for Nvidia. Display-only - separate from trust and ranking.

Company: NVIDIA Corporation·GitHub org profile·Jul 11, 2026
Employees: 11,528·Wikidata (P1128 employees)·Jul 11, 2026
Commercial model: Pure OSS·GitHub org profile (public repos)·Jul 11, 2026

Install

pip install TensorRT-LLM

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

llama.cpp

LLM inference in C/C++

129k★

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

92k★

omlx

LLM inference server with continuous batching and SSD caching for Apple Silicon

22k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

TensorRT LLM offers a Python API for defining Large Language Models and supports optimizations for efficient inference on NVIDIA GPUs. It includes components for creating Python and C++ runtimes for performant inference execution.

Capability facts

Languages: python
Source: github.language+pyproject.toml · Sep 18, 2026

Graph entities

Nvidiacompany

README

Getting Started To get started with TensorRT LLM, visit our documentation: Quick Start Guide Running DeepSeek Installation Guide Supported Hardware, Models, and other Software Benchmarking Performance Release Notes

For agents

This page has a .md twin and JSON over the API.

/tools/nvidia-tensorrt-llm.md GET /api/graphcanon/tools/nvidia-tensorrt-llm GET /api/graphcanon/graph?tool=nvidia-tensorrt-llm