---
title: "lorax"
type: "tool"
slug: "predibase-lorax"
canonical_url: "https://www.graphcanon.com/tools/predibase-lorax"
github_url: "https://github.com/predibase/lorax"
homepage_url: "https://loraexchange.ai"
stars: 3806
forks: 323
primary_language: "Python"
license: "Apache-2.0"
categories: ["inference-serving"]
tags: ["llmops", "llama", "fine-tuning", "lora", "llm", "llm-serving", "gpt", "llm-inference"]
updated_at: "2026-07-07T18:42:10.0173+00:00"
---
# lorax
> Multi-LoRA inference server that scales to serve thousands of fine-tuned LLMs on a single GPU
LoRAX (LoRA eXchange) is an innovative framework for scaling multi-LoRA models, offering dynamic adapter loading, heterogeneous continuous batching, optimized scheduling, and high-performance optimizations.
## Facts
- Repository: https://github.com/predibase/lorax
- Homepage: https://loraexchange.ai
- Stars: 3,806 · Forks: 323 · Open issues: 183 · Watchers: 33
- Primary language: Python
- License: Apache-2.0
- Last pushed: 2026-05-28T18:12:20+00:00
## Categories
- [Inference & Serving](/categories/inference-serving.md)
## Tags
llmops, llama, fine-tuning, lora, llm, llm-serving, gpt, llm-inference
## Related tools
- [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 144,575)
- [vllm](/tools/vllm-project-vllm.md) - A high-throughput and memory-efficient inference and serving engine for LLMs (★ 85,612)
- [rtk](/tools/rtk-ai-rtk.md) - CLI proxy reducing LLM token consumption by 60-90% (★ 69,253)
- [unsloth](/tools/unslothai-unsloth.md) - Unsloth Studio is a web UI for training and running open models locally. (★ 67,883)
- [anything-llm](/tools/mintplex-labs-anything-llm.md) - Stop renting your intelligence. Own it with AnythingLLM. (★ 62,759)
- [litellm](/tools/berriai-litellm.md) - Python SDK and Proxy Server for calling multiple LLM APIs (★ 52,877)
- [ray](/tools/ray-project-ray.md) - Unified framework for scaling AI and Python applications (★ 43,153)
- [khoj](/tools/khoj-ai-khoj.md) - Your AI second brain (★ 35,517)
## README (excerpt)
```text
_LoRAX: Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs_
LoRAX (LoRA eXchange) is a framework that allows users to serve thousands of fine-tuned models on a single GPU, dramatically reducing the cost of serving without compromising on throughput or latency.
## 📖 Table of contents
- [📖 Table of contents](#-table-of-contents)
- [🌳 Features](#-features)
- [🏠 Models](#-models)
- [🏃♂️ Getting Started](#️-getting-started)
- [Requirements](#requirements)
- [Launch LoRAX Server](#launch-lorax-server)
- [Prompt via REST API](#prompt-via-rest-api)
- [Prompt via Python Client](#prompt-via-python-client)
- [Chat via OpenAI API](#chat-via-openai-api)
- [Next steps](#next-steps)
- [🙇 Acknowledgements](#-acknowledgements)
- [🗺️ Roadmap](#️-roadmap)
## 🌳 Features
- 🚅 **Dynamic Adapter Loading:** include any fine-tuned LoRA adapter from [HuggingFace](https://predibase.github.io/lorax/models/adapters/#huggingface-hub), [Predibase](https://predibase.github.io/lorax/models/adapters/#predibase), or [any filesystem](https://predibase.github.io/lorax/models/adapters/#local) in your request, it will be loaded just-in-time without blocking concurrent requests. [Merge adapters](https://predibase.github.io/lorax/guides/merging_adapters/) per request to instantly create powerful ensembles.
- 🏋️♀️ **Heterogeneous Continuous Batching:** packs requests for different adapters together into the same batch, keeping latency and throughput nearly constant with the number of concurrent adapters.
- 🧁 **Adapter Exchange Scheduling:** asynchronously prefetches and offloads adapters between GPU and CPU memory, schedules request batching to optimize the aggregate throughput of the system.
- 👬 **Optimized Inference:** high throughput and low latency optimizations including tensor parallelism, pre-compiled CUDA kernels ([flash-attention](https://arxiv.org/abs/2307.08691), [paged attention](https://arxiv.org/abs/2309.06180), [SGMV](https://arxiv.org/abs/2310.18547)), quantization, token streaming.
- 🚢 **Ready for Production** prebuilt Docker images, Helm charts for Kubernetes, Prometheus metrics, and distributed tracing with Open Telemetry. OpenAI compatible API supporting multi-turn chat conversations. Private adapters through per-request tenant isolation. [Structured Output](https://predibase.github.io/lorax/guides/structured_output) (JSON mode).
- 🤯 **Free for Commercial Use:** Apache 2.0 License. Enough said 😎.
## 🏠 Models
Serving a fine-tuned model with LoRAX consists of two components:
- [Base Model](https://predibase.github.io/lorax/models/base_models): pretrained large model shared across all adapters.
- [Adapter](https://predibase.github.io/lorax/models/adapters): task-specific adapter weights dynamically loaded per request.
LoRAX supports a number of Large Language Models as the base model including [Llama](https://huggingface.co/meta-llama) (including [CodeLlama](https://huggingface.co/codellama)), [Mistral](https://huggingface.co/mistralai) (including [Zephyr](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta)), and [Qwen](https://huggingface.co/Qwen). See [Supported Architectures](https://predibase.github.io/lorax/models/base_models/#supported-architectures) for a complete list of supported base models.
Base models can be loaded in fp16 or quantized with `bitsandbytes`, [GPT-Q](https://arxiv.org/abs/2210.17323), or [AWQ](https://arxiv.org/abs/2306.00978).
Supported adapters include LoRA adapters trained using the [PEFT](https://github.com/huggingface/peft) and [Ludwig](https://ludwig.ai/) libraries. Any of the linear layers in the model can be adapted via LoRA and lo
```
---
**Machine-readable endpoints**
- JSON: [`/api/graphcanon/tools/predibase-lorax`](/api/graphcanon/tools/predibase-lorax)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)
_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_