{"data":{"slug":"nvidia-tensorrt-llm","name":"TensorRT-LLM","tagline":"Python API for defining and optimizing Large Language Models (LLMs) on NVIDIA GPUs","github_url":"https://github.com/NVIDIA/TensorRT-LLM","owner":"NVIDIA","repo":"TensorRT-LLM","owner_avatar_url":"https://avatars.githubusercontent.com/u/1728152?v=4","primary_language":"Python","stars":14091,"forks":2547,"topics":["blackwell","cuda","llm-serving","moe","pytorch"],"archived":false,"github_pushed_at":"2026-07-11T03:06:41+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/nvidia-tensorrt-llm","markdown_url":"https://www.graphcanon.com/tools/nvidia-tensorrt-llm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/nvidia-tensorrt-llm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=nvidia-tensorrt-llm","description":"TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.","homepage_url":"https://nvidia.github.io/TensorRT-LLM","license":"Other","open_issues":1500,"watchers":118,"ai_summary":"TensorRT LLM is designed to enable efficient inference of large language models on NVIDIA GPUs. It offers a user-friendly Python interface, supports state-of-the-art optimizations, and includes components to create high-performance runtimes.","readme_excerpt":"## Getting Started\n\nTo get started with TensorRT-LLM, visit our documentation:\n\n- [Quick Start Guide](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html)\n    - [Running DeepSeek](./examples/models/core/deepseek_v3)\n- [Installation Guide](https://nvidia.github.io/TensorRT-LLM/installation/index.html)\n- [Supported Hardware, Models, and other Software](https://nvidia.github.io/TensorRT-LLM/reference/support-matrix.html)\n- [Benchmarking Performance](https://nvidia.github.io/TensorRT-LLM/performance/performance-tuning-guide/benchmarking-default-performance.html#benchmarking-with-trtllm-bench)\n- [Release Notes](https://nvidia.github.io/TensorRT-LLM/release-notes.html)","github_created_at":"2023-08-16T17:14:27+00:00","created_at":"2026-07-11T10:36:57.350512+00:00","updated_at":"2026-07-12T05:27:06.735466+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"blackwell","name":"blackwell"},{"slug":"cuda","name":"cuda"},{"slug":"llm-serving","name":"llm-serving"},{"slug":"moe","name":"moe"},{"slug":"pytorch","name":"pytorch"}],"trust":{"provenance":{"is_fork":false,"github_id":679366051,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:36:57.946Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":10,"days_since_push":0,"last_release_at":"2026-06-30T01:11:27Z"},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":16,"high_count":0,"last_scan_at":"2026-07-11T10:36:58.870Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T15:09:42.848Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-11T15:09:42.848Z"},"license_spdx":{"value":"Other","source":"github.license","observed_at":"2026-07-11T15:09:42.848Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"oss","summary":"Open source software (OSS) available under a license other than those listed in common OSS categories, implying free use but potentially with restrictions."},"requirements":{"notes":["NVIDIA GPU hardware is required for the tool to take full advantage of its optimization capabilities."]},"constraints":{"pricing_model":"oss"},"when_to_use":["When you are developing or deploying large language models (LLMs) specifically on NVIDIA GPU hardware.","If your project requires state-of-the-art performance tuning tailored to LLMs with integration for PyTorch and CUDA.","For scenarios that need both Python and C++ runtimes orchestrating the inference, providing flexibility in deployment."],"when_not_to_use":["When working on CPUs or non-NVIDIA GPUs as the optimizations and hardware support are NVIDIA-specific.","If you prioritize portability across different frameworks over high-performance tuning since TensorRT LLM is tightly integrated with NVIDIA technologies.","For projects that do not require deep level performance optimizations and prefer more general-purpose serving solutions."],"source":"enrich:decision_facts","observed_at":"2026-07-11T15:10:13.287Z"},"constraint_facets":{"pricing_model":"oss"},"decision_summary":[{"label":"Pricing","value":"oss - Open source software (OSS) available under a license other than those listed in common OSS categories, implying free use but potentially with restrictions."},{"label":"Requirements","value":"NVIDIA GPU hardware is required for the tool to take full advantage of its optimization capabilities."},{"label":"Adopt for","value":"`TensorRT LLM` is a specialized Python API for optimizing and efficiently running large language models on NVIDIA GPUs, featuring user-friendly interfaces and high-performance optimizations."}]}}