Results for “llm-serving”

vllm

vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

ray

ray-project/ray

Unified framework for scaling AI and Python applications

llm-action

liguodongiot/llm-action

分享大模型技术原理与实战经验，涵盖工程化和应用落地

OpenLLM

bentoml/OpenLLM

Self-Hosting LLMs Made Easy

BentoML

bentoml/BentoML

The easiest way to serve AI apps and models

superduper

superduper-io/superduper

End-to-end framework for building custom AI applications and agents.

lorax

predibase/lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

vllm-ascend

vllm-project/vllm-ascend

Community maintained hardware plugin for vLLM on Ascend

aici

microsoft/aici

Artificial Intelligence Controller Interface (AICI)

rtp-llm

alibaba/rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.