GraphCanon updated 2w · GitHub synced 2w · 58 views this month
Decision brief
Use OpenLLM for easy deployment of a wide range of open-source LLMs through an OpenAI-compatible API with support for cloud environments and fine-tuning.
Good fit when
- You require OpenAI-compatible APIs to serve a diverse set of state-of-the-art open-source LLMs, such as DeepSeek, Llama, or Qwen2.5, in both local and cloud deployment scenarios.
- Your use case involves the need for an integrated chat UI along with the ability to run models natively without storing weights locally, thereby reducing storage costs.
Avoid when
- If your project primarily focuses on proprietary models that are not open-source and you do not want to convert or migrate them to an OpenAI-compatible API.
- In situations where direct model weight management is required for compliance or security reasons, as OpenLLM does not store the model weights.
Observed Jul 11, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Very active (3d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install OpenLLMHow it fits your stack(6)
Typed graph edges - alternatives, integrations, successors, and dependencies. Ranked by relationship type, not raw GitHub stars.
Alternative
Integrates
Relationship graph
Optional deeper exploration of typed edges and category neighbours.
Similar tools
Same-category neighbours not already linked as typed edges.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
OpenLLM is a tool that allows developers to deploy various open-source large language models (LLMs) with ease by serving them through an OpenAI-compatible API interface. It supports a wide array of models, including customization options and comes with deployment features for cloud environments like Docker and Kubernetes.
Capability facts
- CLI
- CLI entrypoint
Source: pyproject.toml:[project.scripts] · Aug 7, 2026
- Languages
- python
Source: github.language+pyproject.toml · Aug 7, 2026
Categories
Graph entities
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 7, 2026)
pip install openllm # or pip3 install openllmSource link
Tags
README
🦾 OpenLLM: Self-Hosting LLMs Made Easy
OpenLLM allows developers to run any open-source LLMs (Llama 3.3, Qwen2.5, Phi3 and more) or custom models as OpenAI-compatible APIs with a single command. It features a built-in chat UI, state-of-the-art inference backends, and a simplified workflow for creating enterprise-grade cloud deployment with Docker, Kubernetes, and BentoCloud.
Understand the design philosophy of OpenLLM.
Get Started
Run the following commands to install OpenLLM and explore it interactively.
pip install openllm # or pip3 install openllm
openllm hello
Supported models
OpenLLM supports a wide range of state-of-the-art open-source LLMs. You can also add a model repository to run custom models with OpenLLM.
| Model | Parameters | Required GPU | Start a Server |
|---|---|---|---|
| deepseek | r1-671b | 80Gx16 | openllm serve deepseek:r1-671b |
| gemma2 | 2b | 12G | openllm serve gemma2:2b |
| gemma3 | 3b | 12G | openllm serve gemma3:3b |
| jamba1.5 | mini-ff0a | 80Gx2 | openllm serve jamba1.5:mini-ff0a |
| llama3.1 | 8b | 24G | openllm serve llama3.1:8b |
| llama3.2 | 1b | 24G | openllm serve llama3.2:1b |
| llama3.3 | 70b | 80Gx2 | openllm serve llama3.3:70b |
| llama4 | 17b16e | 80Gx8 | openllm serve llama4:17b16e |
| mistral | 8b-2410 | 24G | openllm serve mistral:8b-2410 |
| mistral-large | 123b-2407 | 80Gx4 | openllm serve mistral-large:123b-2407 |
| phi4 | 14b | 80G | openllm serve phi4:14b |
| pixtral | 12b-2409 | 80G | openllm serve pixtral:12b-2409 |
| qwen2.5 | 7b | 24G | openllm serve qwen2.5:7b |
| qwen2.5-coder | 3b | 24G | openllm serve qwen2.5-coder:3b |
| qwq | 32b | 80G | openllm serve qwq:32b |
For the full model list, see the OpenLLM models repository.
Start an LLM server
To start an LLM server locally, use the openllm serve command and specify the model version.
[!NOTE] OpenLLM does not store model weights. A Hugging Face token (HF_TOKEN) is required for gated models.
- Create your Hugging Face token here.
- Request access to the gated model, such as meta-llama/Llama-3.2-1B-Instruct.
- Set your token as an environment variable by running:
export HF_TOKEN=<your token>
openllm serve llama3.2:1b
The server will be accessible at http://localhost:3000, providing OpenAI-compatible APIs for interaction. You can call the endpoints with different frameworks and tools that support OpenAI-compatible APIs. Typically, you may need to specify
For agents
This page has a .md twin and JSON over the API.