“model-serving”

A high-throughput and memory-efficient inference and serving engine for LLMs

The easiest way to serve AI apps and models

Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Community maintained hardware plugin for vLLM on Ascend

Reproducible development environment for humans and agents

Artificial Intelligence Controller Interface (AICI)

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.