{"data":{"slug":"triton-inference-server-server","name":"server","tagline":"The Triton Inference Server provides an optimized cloud and edge inferencing solution.","github_url":"https://github.com/triton-inference-server/server","owner":"triton-inference-server","repo":"server","owner_avatar_url":"https://avatars.githubusercontent.com/u/68086070?v=4","primary_language":"Python","stars":10822,"forks":1806,"topics":["cloud","datacenter","deep-learning","edge","gpu","inference","machine-learning"],"archived":false,"github_pushed_at":"2026-07-11T14:49:10+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/triton-inference-server-server","markdown_url":"https://www.graphcanon.com/tools/triton-inference-server-server.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/triton-inference-server-server","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=triton-inference-server-server","description":"The Triton Inference Server provides an optimized cloud and edge inferencing solution. ","homepage_url":"https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html","license":"BSD-3-Clause","open_issues":901,"watchers":145,"ai_summary":null,"readme_excerpt":">[!WARNING]\n>You are currently on the `main` branch which tracks under-development progress\n>towards the next release. The current release is version [2.70.0](https://github.com/triton-inference-server/server/releases/latest)\n>and corresponds to the 26.06 container release on NVIDIA GPU Cloud (NGC).\n\n# Triton Inference Server\n\nTriton Inference Server is an open source inference serving software that\nstreamlines AI inferencing. Triton enables teams to deploy any AI model from\nmultiple deep learning and machine learning frameworks, including TensorRT,\nPyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. Triton\nInference Server supports inference across cloud, data center, edge and embedded\ndevices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia. Triton Inference\nServer delivers optimized performance for many query types, including real time,\nbatched, ensembles and audio/video streaming. Triton inference Server is part of\n[NVIDIA AI Enterprise](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/),\na software platform that accelerates the data science pipeline and streamlines\nthe development and deployment of production AI.\n\nMajor features include:\n\n- [Supports multiple deep learning\n  frameworks](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton)\n- [Supports multiple machine learning\n  frameworks](https://github.com/triton-inference-server/fil_backend)\n- [Concurrent model\n  execution](docs/user_guide/architecture.md#concurrent-model-execution)\n- [Dynamic batching](docs/user_guide/batcher.md#dynamic-batcher)\n- [Sequence batching](docs/user_guide/batcher.md#sequence-batcher) and\n  [implicit state management](docs/user_guide/architecture.md#implicit-state-management)\n  for stateful models\n- Provides [Backend API](https://github.com/triton-inference-server/backend) that\n  allows adding custom backends and pre/post processing operations\n- Supports writing custom backends in python, a.k.a.\n  [Python-based backends.](https://github.com/triton-inference-server/backend/blob/main/docs/python_based_backends.md#python-based-backends)\n- Model pipelines using\n  [Ensembling](docs/user_guide/architecture.md#ensemble-models) or [Business\n  Logic Scripting\n  (BLS)](https://github.com/triton-inference-server/python_backend#business-logic-scripting)\n- [HTTP/REST and GRPC inference\n  protocols](docs/customization_guide/inference_protocols.md) based on the community\n  developed [KServe\n  protocol](https://github.com/kserve/kserve/tree/master/docs/predict-api/v2)\n- A [C API](docs/customization_guide/inprocess_c_api.md) and\n  [Java API](docs/customization_guide/inprocess_java_api.md)\n  allow Triton to link directly into your application for edge and other in-process use cases\n- [Metrics](docs/user_guide/metrics.md) indicating GPU utilization, server\n  throughput, server latency, and more\n\n**New to Triton Inference Server?** Make use of\n[these tutorials](https://github.com/triton-inference-server/tutorials)\nto begin your Triton journey!\n\nJoin the [Triton and TensorRT community](https://www.nvidia.com/en-us/deep-learning-ai/triton-tensorrt-newsletter/) and\nstay current on the latest product updates, bug fixes, content, best practices,\nand more.  Need enterprise support?  NVIDIA global support is available for Triton\nInference Server with the\n[NVIDIA AI Enterprise software suite](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/).\n\n## Serve a Model in 3 Easy Steps\n\n```bash\n# Step 1: Create the example model repository\ngit clone -b r26.06 https://github.com/triton-inference-server/server.git\ncd server/docs/examples\n./fetch_models.sh\n\n# Step 2: Launch triton from the NGC Triton container\ndocker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:26.06-py3 tritonserver --model-repository=/models --model-control-mode explicit --load-model densenet_onnx\n\n# Step 3: Sending an Inference Request\n# In a separate console, launch t","github_created_at":"2018-10-04T21:10:30+00:00","created_at":"2026-07-11T23:12:50.321029+00:00","updated_at":"2026-07-11T23:13:00.914258+00:00","categories":[{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"},{"slug":"speech-audio","name":"Speech & Audio","url":"https://www.graphcanon.com/categories/speech-audio","markdown_url":"https://www.graphcanon.com/categories/speech-audio.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/speech-audio"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"deep-learning","name":"deep-learning"},{"slug":"gpu","name":"gpu"},{"slug":"machine-learning","name":"machine-learning"},{"slug":"datacenter","name":"datacenter"},{"slug":"python","name":"python"},{"slug":"edge","name":"edge"},{"slug":"cloud","name":"cloud"},{"slug":"inference","name":"inference"}],"trust":{"provenance":{"is_fork":false,"github_id":151636194,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:12:52.075Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":3,"days_since_push":0,"last_release_at":"2026-06-26T18:41:12Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:12:52.599Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:12:51.795Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-11T23:12:51.795Z"},"license_spdx":{"value":"BSD-3-Clause","source":"github.license","observed_at":"2026-07-11T23:12:51.795Z"}}}}