“llm-serving”

A high-throughput and memory-efficient inference and serving engine for LLMs

Unified framework for scaling AI and Python applications

分享大模型技术原理与实战经验，涵盖工程化和应用落地

Self-Hosting LLMs Made Easy

The easiest way to serve AI apps and models

End-to-end framework for building custom AI applications and agents.

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Community maintained hardware plugin for vLLM on Ascend

Artificial Intelligence Controller Interface (AICI)

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.