{"data":{"slug":"triton-inference-server-server","name":"server","tagline":"Optimized cloud and edge inferencing solution","github_url":"https://github.com/triton-inference-server/server","owner":"triton-inference-server","repo":"server","owner_avatar_url":"https://avatars.githubusercontent.com/u/68086070?v=4","primary_language":"Python","stars":10995,"forks":1837,"topics":["cloud","datacenter","deep-learning","edge","gpu","inference","machine-learning"],"archived":false,"github_pushed_at":"2026-09-17T14:35:44+00:00","maintenance_label":"Very active","stars_delta_30d":110,"url":"https://www.graphcanon.com/tools/triton-inference-server-server","markdown_url":"https://www.graphcanon.com/tools/triton-inference-server-server.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/triton-inference-server-server","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=triton-inference-server-server","description":"The Triton Inference Server provides an optimized cloud and edge inferencing solution. ","homepage_url":"https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html","license":"BSD-3-Clause","open_issues":894,"watchers":146,"ai_summary":"Triton Inference Server is an open-source software for streamlining AI inferencing, supporting multiple deep learning and machine learning frameworks, and enabling deployment across various devices and environments.","readme_excerpt":">[!WARNING]\n>You are currently on the `main` branch which tracks under-development progress\n>towards the next release. The current release is version [2.72.0](https://github.com/triton-inference-server/server/releases/latest)\n>and corresponds to the 26.08 container release on NVIDIA GPU Cloud (NGC).\n\n# Triton Inference Server\n\nTriton Inference Server is an open source inference serving software that\nstreamlines AI inferencing. Triton enables teams to deploy any AI model from\nmultiple deep learning and machine learning frameworks, including TensorRT,\nPyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. Triton\nInference Server supports inference across cloud, data center, edge and embedded\ndevices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia. Triton Inference\nServer delivers optimized performance for many query types, including real time,\nbatched, ensembles and audio/video streaming. Triton inference Server is part of\n[NVIDIA AI Enterprise](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/),\na software platform that accelerates the data science pipeline and streamlines\nthe development and deployment of production AI.\n\nMajor features include:\n\n- [Supports multiple deep learning\n  frameworks](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton)\n- [Supports multiple machine learning\n  frameworks](https://github.com/triton-inference-server/fil_backend)\n- [Concurrent model\n  execution](docs/user_guide/architecture.md#concurrent-model-execution)\n- [Dynamic batching](docs/user_guide/batcher.md#dynamic-batcher)\n- [Sequence batching](docs/user_guide/batcher.md#sequence-batcher) and\n  [implicit state management](docs/user_guide/architecture.md#implicit-state-management)\n  for stateful models\n- Provides [Backend API](https://github.com/triton-inference-server/backend) that\n  allows adding custom backends and pre/post processing operations\n- Supports writing custom backends in python, a.k.a.\n  [Python-based backends.](https://github.com/triton-inference-server/backend/blob/main/docs/python_based_backends.md#python-based-backends)\n- Model pipelines using\n  [Ensembling](docs/user_guide/architecture.md#ensemble-models) or [Business\n  Logic Scripting\n  (BLS)](https://github.com/triton-inference-server/python_backend#business-logic-scripting)\n- [HTTP/REST and GRPC inference\n  protocols](docs/customization_guide/inference_protocols.md) based on the community\n  developed [KServe\n  protocol](https://github.com/kserve/kserve/tree/master/docs/predict-api/v2)\n- A [C API](docs/customization_guide/inprocess_c_api.md) and\n  [Java API](docs/customization_guide/inprocess_java_api.md)\n  allow Triton to link directly into your application for edge and other in-process use cases\n- [Metrics](docs/user_guide/metrics.md) indicating GPU utilization, server\n  throughput, server latency, and more\n\n**New to Triton Inference Server?** Make use of\n[these tutorials](https://github.com/triton-inference-server/tutorials)\nto begin your Triton journey!\n\nJoin the [Triton and TensorRT community](https://www.nvidia.com/en-us/deep-learning-ai/triton-tensorrt-newsletter/) and\nstay current on the latest product updates, bug fixes, content, best practices,\nand more.  Need enterprise support?  NVIDIA global support is available for Triton\nInference Server with the\n[NVIDIA AI Enterprise software suite](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/).\n\n## Serve a Model in 3 Easy Steps\n\n```bash\n# Step 1: Create the example model repository\ngit clone -b r26.08 https://github.com/triton-inference-server/server.git\ncd server/docs/examples\n./fetch_models.sh\n\n# Step 2: Launch triton from the NGC Triton container\ndocker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:26.08-py3 tritonserver --model-repository=/models --model-control-mode explicit --load-model densenet_onnx\n\n# Step 3: Sending an Inference Request\n# In a separate console, launch t","github_created_at":"2018-10-04T21:10:30+00:00","created_at":"2026-07-11T23:12:50.321029+00:00","updated_at":"2026-09-20T04:30:00.375166+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"cloud","name":"cloud"},{"slug":"datacenter","name":"datacenter"},{"slug":"deep-learning","name":"deep-learning"},{"slug":"edge","name":"edge"},{"slug":"gpu","name":"gpu"},{"slug":"inference","name":"inference"},{"slug":"machine-learning","name":"machine-learning"},{"slug":"nvidia","name":"nvidia"}],"trust":{"provenance":{"is_fork":false,"github_id":151636194,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:20:21.124Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":3,"days_since_push":0,"last_release_at":"2026-08-31T19:07:38Z","stars_delta_30d":110,"open_issues_delta_30d":-11},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:20:23.306Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:20:20.042Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-18T08:20:20.042Z"},"license_spdx":{"value":"BSD-3-Clause","source":"github.license","observed_at":"2026-09-18T08:20:20.042Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["Use Triton Inference Server when you need to deploy AI models across multiple frameworks such as TensorRT, PyTorch, ONNX, and OpenVINO.","Opt for Triton Inference Server if you require optimized performance for real-time, batched, and ensemble queries, as well as audio/video streaming.","Choose Triton Inference Server when you need to support inference on NVIDIA GPUs, x86, and ARM CPUs, or AWS Inferentia.","Use Triton Inference Server if you are working within the NVIDIA AI Enterprise ecosystem and require enterprise support."],"when_not_to_use":["Avoid Triton Inference Server if your project is not compatible with the NVIDIA AI Enterprise ecosystem and you do not require support for NVIDIA GPUs.","Do not use Triton Inference Server if your deployment environment does not support the frameworks it is optimized for, such as TensorRT, PyTorch, ONNX, and OpenVINO.","If your application does not require dynamic batching or sequence batching, Triton Inference Server might not be the best choice.","If you are looking for a solution that does not integrate with the KServe protocol for HTTP/REST and GRPC inference, Triton Inference Server may not be suitable."],"source":"enrich:decision_facts","observed_at":"2026-09-18T10:41:33.123Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Triton Inference Server is an open-source solution for streamlining AI inferencing, supporting multiple deep learning and machine learning frameworks, and enabling deployment across various devices and environments."}]}}