Results for “model-serving”

vllm

vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

BentoML

bentoml/BentoML

The easiest way to serve AI apps and models

AI-Infra-from-Zero-to-Hero

HuaizhengZhang/AI-Infra-from-Zero-to-Hero

Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice

lorax

predibase/lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

vllm-ascend

vllm-project/vllm-ascend

Community maintained hardware plugin for vLLM on Ascend

envd

tensorchord/envd

Reproducible development environment for humans and agents

aici

microsoft/aici

Prompts as (Wasm) Programs

rtp-llm

alibaba/rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine