{"data":{"slug":"bentoml-openllm","name":"OpenLLM","tagline":"Run any open-source LLMs as OpenAI compatible API endpoint in the cloud.","github_url":"https://github.com/bentoml/OpenLLM","owner":"bentoml","repo":"OpenLLM","owner_avatar_url":"https://avatars.githubusercontent.com/u/49176046?v=4","primary_language":"Python","stars":12454,"forks":828,"topics":["bentoml","fine-tuning","llama","llama2","llama3-1","llama3-2","llama3-2-vision","llm","llm-inference","llm-ops","llm-serving","llmops","mistral","mlops","model-inference","open-source-llm","openllm","vicuna"],"archived":false,"github_pushed_at":"2026-08-03T16:59:03+00:00","maintenance_label":"Very active","stars_delta_30d":66,"url":"https://www.graphcanon.com/tools/bentoml-openllm","markdown_url":"https://www.graphcanon.com/tools/bentoml-openllm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/bentoml-openllm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=bentoml-openllm","description":"Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.","homepage_url":"https://bentoml.com","license":"Apache-2.0","open_issues":18,"watchers":82,"ai_summary":"OpenLLM is a tool that allows developers to deploy various open-source large language models (LLMs) with ease by serving them through an OpenAI-compatible API interface. It supports a wide array of models, including customization options and comes with deployment features for cloud environments like Docker and Kubernetes.","readme_excerpt":"
\n\n

🦾 OpenLLM: Self-Hosting LLMs Made Easy

\n\n\n\n\n\n\n\n
\n\nOpenLLM allows developers to run **any open-source LLMs** (Llama 3.3, Qwen2.5, Phi3 and [more](#supported-models)) or **custom models** as **OpenAI-compatible APIs** with a single command. It features a [built-in chat UI](#chat-ui), state-of-the-art inference backends, and a simplified workflow for creating enterprise-grade cloud deployment with Docker, Kubernetes, and [BentoCloud](#deploy-to-bentocloud).\n\nUnderstand the [design philosophy of OpenLLM](https://www.bentoml.com/blog/from-ollama-to-openllm-running-llms-in-the-cloud).\n\n## Get Started\n\nRun the following commands to install OpenLLM and explore it interactively.\n\n```bash\npip install openllm # or pip3 install openllm\nopenllm hello\n```\n\n\n\n## Supported models\n\nOpenLLM supports a wide range of state-of-the-art open-source LLMs. You can also add a [model repository to run custom models](#set-up-a-custom-repository) with OpenLLM.\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
ModelParametersRequired GPUStart a Server
deepseekr1-671b80Gx16openllm serve deepseek:r1-671b
gemma22b12Gopenllm serve gemma2:2b
gemma33b12Gopenllm serve gemma3:3b
jamba1.5mini-ff0a80Gx2openllm serve jamba1.5:mini-ff0a
llama3.18b24Gopenllm serve llama3.1:8b
llama3.21b24Gopenllm serve llama3.2:1b
llama3.370b80Gx2openllm serve llama3.3:70b
llama417b16e80Gx8openllm serve llama4:17b16e
mistral8b-241024Gopenllm serve mistral:8b-2410
mistral-large123b-240780Gx4openllm serve mistral-large:123b-2407
phi414b80Gopenllm serve phi4:14b
pixtral12b-240980Gopenllm serve pixtral:12b-2409
qwen2.57b24Gopenllm serve qwen2.5:7b
qwen2.5-coder3b24Gopenllm serve qwen2.5-coder:3b
qwq32b80Gopenllm serve qwq:32b
\n\nFor the full model list, see the [OpenLLM models repository](https://github.com/bentoml/openllm-models).\n\n## Start an LLM server\n\nTo start an LLM server locally, use the `openllm serve` command and specify the model version.\n\n> [!NOTE]\n> OpenLLM does not store model weights. A Hugging Face token (HF_TOKEN) is required for gated models.\n>\n> 1. Create your Hugging Face token [here](https://huggingface.co/settings/tokens).\n> 2. Request access to the gated model, such as [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct).\n> 3. Set your token as an environment variable by running:\n> ```bash\n> export HF_TOKEN=\n> ```\n\n```bash\nopenllm serve llama3.2:1b\n```\n\nThe server will be accessible at [http://localhost:3000](http://localhost:3000/), providing OpenAI-compatible APIs for interaction. You can call the endpoints with different frameworks and tools that support OpenAI-compatible APIs. Typically, you may need to specify","github_created_at":"2023-04-19T00:27:52+00:00","created_at":"2026-07-07T17:41:39.564174+00:00","updated_at":"2026-08-07T12:00:40.979508+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"bentoml","name":"bentoml"},{"slug":"fine-tuning","name":"fine-tuning"},{"slug":"llama","name":"llama"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"llm-serving","name":"llm-serving"},{"slug":"open-source-llm","name":"open-source-llm"}],"trust":{"provenance":{"is_fork":false,"github_id":629749002,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-07T12:00:40.173Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":3,"last_release_at":"2025-04-21T04:10:55Z","stars_delta_30d":66,"open_issues_delta_30d":1},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:37:36.598Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-07T12:00:40.636Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-08-07T12:00:40.636Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-07T12:00:40.636Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-07T12:00:40.636Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["You require OpenAI-compatible APIs to serve a diverse set of state-of-the-art open-source LLMs, such as DeepSeek, Llama, or Qwen2.5, in both local and cloud deployment scenarios.","Your use case involves the need for an integrated chat UI along with the ability to run models natively without storing weights locally, thereby reducing storage costs."],"when_not_to_use":["If your project primarily focuses on proprietary models that are not open-source and you do not want to convert or migrate them to an OpenAI-compatible API.","In situations where direct model weight management is required for compliance or security reasons, as OpenLLM does not store the model weights."],"source":"enrich:decision_facts","observed_at":"2026-07-11T15:25:40.906Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Use OpenLLM for easy deployment of a wide range of open-source LLMs through an OpenAI-compatible API with support for cloud environments and fine-tuning."}]}}