“vllm”

Community maintained hardware plugin for vLLM on Ascend

A high-throughput and memory-efficient inference and serving engine for LLMs

An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray

Python SDK and Proxy Server for calling over 100 LLM APIs in OpenAI format

Less Code, Lower Barrier, Faster Deployment

Evaluate your LLM's response with Prometheus and GPT4