“tensorrt-llm”

TensorRT-LLM

Python API for defining and optimizing Large Language Models (LLMs) on NVIDIA GPUs

ome

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

dynamo

A Datacenter Scale Distributed Inference Serving Framework

Awesome-LLM-Inference

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

WhisperLive

A nearly-live implementation of OpenAI's Whisper.