lorax
predibase/lorax
Multi-LoRA inference server that scales to serve thousands of fine-tuned LLMs on a single GPU
Overview
LoRAX (LoRA eXchange) is an innovative framework for scaling multi-LoRA models, offering dynamic adapter loading, heterogeneous continuous batching, optimized scheduling, and high-performance optimizations.
Categories
Tags
Similar tools
open-webui
open-webui/open-webui
User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
vllm
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
rtk
rtk-ai/rtk
CLI proxy reducing LLM token consumption by 60-90%
unsloth
unslothai/unsloth
Unsloth Studio is a web UI for training and running open models locally.
anything-llm
Mintplex-Labs/anything-llm
Stop renting your intelligence. Own it with AnythingLLM.
litellm
BerriAI/litellm
Python SDK and Proxy Server for calling multiple LLM APIs
Install
pip install loraxREADME
LoRAX: Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
LoRAX (LoRA eXchange) is a framework that allows users to serve thousands of fine-tuned models on a single GPU, dramatically reducing the cost of serving without compromising on throughput or latency.
📖 Table of contents
- 📖 Table of contents
- 🌳 Features
- 🏠 Models
- 🏃♂️ Getting Started
- Requirements
- Launch LoRAX Server
- Prompt via REST API
- Prompt via Python Client
- Chat via OpenAI API
- Next steps
- 🙇 Acknowledgements
- 🗺️ Roadmap
🌳 Features
- 🚅 Dynamic Adapter Loading: include any fine-tuned LoRA adapter from HuggingFace, Predibase, or any filesystem in your request, it will be loaded just-in-time without blocking concurrent requests. Merge adapters per request to instantly create powerful ensembles.
- 🏋️♀️ Heterogeneous Continuous Batching: packs requests for different adapters together into the same batch, keeping latency and throughput nearly constant with the number of concurrent adapters.
- 🧁 Adapter Exchange Scheduling: asynchronously prefetches and offloads adapters between GPU and CPU memory, schedules request batching to optimize the aggregate throughput of the system.
- 👬 Optimized Inference: high throughput and low latency optimizations including tensor parallelism, pre-compiled CUDA kernels (flash-attention, paged attention, SGMV), quantization, token streaming.
- 🚢 Ready for Production prebuilt Docker images, Helm charts for Kubernetes, Prometheus metrics, and distributed tracing with Open Telemetry. OpenAI compatible API supporting multi-turn chat conversations. Private adapters through per-request tenant isolation. Structured Output (JSON mode).
- 🤯 Free for Commercial Use: Apache 2.0 License. Enough said 😎.
🏠 Models
Serving a fine-tuned model with LoRAX consists of two components:
- Base Model: pretrained large model shared across all adapters.
- Adapter: task-specific adapter weights dynamically loaded per request.
LoRAX supports a number of Large Language Models as the base model including Llama (including CodeLlama), Mistral (including Zephyr), and Qwen. See Supported Architectures for a complete list of supported base models.
Base models can be loaded in fp16 or quantized with bitsandbytes, GPT-Q, or AWQ.
Supported adapters include LoRA adapters trained using the PEFT and Ludwig libraries. Any of the linear layers in the model can be adapted via LoRA and lo