OpenLLM allows developers to run any open-source LLMs (Llama 3.3, Qwen2.5, Phi3 and more) or custom models as OpenAI-compatible APIs with a single command. It features a built-in chat UI, state-of-the-art inference backends, and a simplified workflow for creating enterprise-grade cloud deployment with Docker, Kubernetes, and BentoCloud.

Understand the design philosophy of OpenLLM.

Get Started

Run the following commands to install OpenLLM and explore it interactively.

pip install openllm  # or pip3 install openllm
openllm hello

Supported models

OpenLLM supports a wide range of state-of-the-art open-source LLMs. You can also add a model repository to run custom models with OpenLLM.

Model	Parameters	Required GPU	Start a Server
deepseek	r1-671b	80Gx16	openllm serve deepseek:r1-671b
gemma2	2b	12G	openllm serve gemma2:2b
gemma3	3b	12G	openllm serve gemma3:3b
jamba1.5	mini-ff0a	80Gx2	openllm serve jamba1.5:mini-ff0a
llama3.1	8b	24G	openllm serve llama3.1:8b
llama3.2	1b	24G	openllm serve llama3.2:1b
llama3.3	70b	80Gx2	openllm serve llama3.3:70b
llama4	17b16e	80Gx8	openllm serve llama4:17b16e
mistral	8b-2410	24G	openllm serve mistral:8b-2410
mistral-large	123b-2407	80Gx4	openllm serve mistral-large:123b-2407
phi4	14b	80G	openllm serve phi4:14b
pixtral	12b-2409	80G	openllm serve pixtral:12b-2409
qwen2.5	7b	24G	openllm serve qwen2.5:7b
qwen2.5-coder	3b	24G	openllm serve qwen2.5-coder:3b
qwq	32b	80G	openllm serve qwq:32b

For the full model list, see the OpenLLM models repository.

Start an LLM server

To start an LLM server locally, use the openllm serve command and specify the model version.

[!NOTE] OpenLLM does not store model weights. A Hugging Face token (HF_TOKEN) is required for gated models.

1. Create your Hugging Face token here.
2. Request access to the gated model, such as meta-llama/Llama-3.2-1B-Instruct.
3. Set your token as an environment variable by running:

   export HF_TOKEN=<your token>
   

openllm serve llama3.2:1b

The server will be accessible at http://localhost:3000, providing OpenAI-compatible APIs for interaction. You can call the endpoints with different frameworks and tools that support OpenAI-compatible APIs. Typically, you may need to specify