GraphCanon updated 4d · GitHub synced 4d
Decision brief
optillm is an optimizing inference proxy for LLMs that provides enhanced deployment options through Docker, supporting both full and lightweight configurations.
Good fit when
- Use optillm when you require automatic optimization of the server approach to enhance reasoning capabilities with large language models.
- Consider opting for optillm if your project involves agents or complex workflows managed via API gateway services.
Avoid when
- Avoid optillm when your application does not require proxy server optimization for large language models; simpler serving setups may suffice.
- Do not use optillm if your deployment environment strictly prohibits the use of Docker images or containers, given that this tool heavily relies on Docker for its various configurations.
- Hosting:
- self hosted - This open-source proxy supports diverse hosting environments and can be run via Docker for flexibility in deployment.
- Pricing:
- freemium - optillm is available under the Apache-2.0 license, which makes it free to use and distribute without cost.
Observed Jul 14, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Steady (30d since push)
- As of 4d
- Provenance
- Not a fork · Organization account
- As of 4d
- Security (OSV)
- 1 low (1 low)
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install optillmHow it fits your stack(7)
Typed graph edges - alternatives, integrations, successors, and dependencies. Ranked by relationship type, not raw GitHub stars.
Alternative
Integrates
Relationship graph
Optional deeper exploration of typed edges and category neighbours.
Similar tools
Same-category neighbours not already linked as typed edges.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
A Python-based tool that optimizes the proxy servers for large language models (LLMs) and supports various deployment options including Docker.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Aug 17, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Aug 17, 2026
- CLI
- CLI entrypoint
Source: pyproject.toml:[project.scripts] · Aug 17, 2026
- Languages
- python
Source: github.language+pyproject.toml · Aug 17, 2026
Categories
Graph entities
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 17, 2026)
tory and add any environment variables you want to set. For example, to set the OpenAI API key, add the following line to the `.env` file:Source link
Tags
README
🚀 Quick Start
Get powerful reasoning improvements in 3 simple steps:
---
### Using docker
```bash
docker pull ghcr.io/algorithmicsuperintelligence/optillm:latest
docker run -p 8000:8000 ghcr.io/algorithmicsuperintelligence/optillm:latest
2024-10-22 07:45:05,612 - INFO - Loaded plugin: privacy
2024-10-22 07:45:06,293 - INFO - Loaded plugin: memory
2024-10-22 07:45:06,293 - INFO - Starting server with approach: auto
Available Docker image variants:
- Full image (
latest): Includes all dependencies for local inference and plugins - Proxy-only (
latest-proxy): Lightweight image without local inference capabilities - Offline (
latest-offline): Self-contained image with pre-downloaded models (spaCy) for fully offline operation
---
### Install from source
Clone the repository with `git` and use `pip install` to setup the dependencies.
```bash
git clone https://github.com/algorithmicsuperintelligence/optillm.git
cd optillm
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
Running with Docker
optillm can optionally be built and run using Docker and the provided Dockerfile.
Using Docker Compose
-
Make sure you have Docker and Docker Compose installed on your system.
-
Either update the environment variables in the docker-compose.yaml file or create a
.envfile in the project root directory and add any environment variables you want to set. For example, to set the OpenAI API key, add the following line to the.envfile:OPENAI_API_KEY=your_openai_api_key_here -
Run the following command to start optillm:
docker compose up -dThis will build the Docker image if it doesn't exist and start the optillm service.
-
optillm will be available at
http://localhost:8000.
When using Docker, you can set these parameters as environment variables. For example, to set the approach and model, you would use:
OPTILLM_APPROACH=mcts
OPTILLM_MODEL=gpt-4
To secure the optillm proxy with an API key, set the OPTILLM_API_KEY environment variable:
OPTILLM_API_KEY=your_secret_api_key
When the API key is set, clients must include it in their requests using the Authorization header:
Authorization: Bearer your_secret_api_key
For agents
This page has a .md twin and JSON over the API.