Results for “inference & serving”

vllm

vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

BentoML

bentoml/BentoML

The easiest way to serve AI apps and models

lorax

predibase/lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

ray

ray-project/ray

Unified framework for scaling AI and Python applications

llm-action

liguodongiot/llm-action

分享大模型技术原理与实战经验，涵盖工程化和应用落地

OpenLLM

bentoml/OpenLLM

Self-Hosting LLMs Made Easy

superduper

superduper-io/superduper

End-to-end framework for building custom AI applications and agents.

aici

microsoft/aici

Prompts as (Wasm) Programs

vllm-ascend

vllm-project/vllm-ascend

Community maintained hardware plugin for vLLM on Ascend

AI-Infra-from-Zero-to-Hero

HuaizhengZhang/AI-Infra-from-Zero-to-Hero

Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice

PowerInfer

Tiiny-AI/PowerInfer

High-speed Large Language Model Serving for Local Deployment

rtp-llm

alibaba/rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine

sglang

sgl-project/sglang

SGLang high-performance serving framework

pixeltable

pixeltable/pixeltable

Unified multimodal backend for AI data apps

paddler

intentee/paddler

LLM load balancer and serving platform for self-hosting LLMs/VLMs