3 tools · Python
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
bentoml/BentoML
The easiest way to serve AI apps and models
predibase/lorax
Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
