{"data":{"slug":"algorithmicsuperintelligence-optillm","name":"optillm","tagline":"Optimizing inference proxy for LLMs","github_url":"https://github.com/algorithmicsuperintelligence/optillm","owner":"algorithmicsuperintelligence","repo":"optillm","owner_avatar_url":"https://avatars.githubusercontent.com/u/238764598?v=4","primary_language":"Python","stars":4244,"forks":385,"topics":["agent","agentic-ai","agentic-framework","agentic-workflow","agents","api-gateway","chain-of-thought","genai","large-language-models","llm","llm-inference","llmapi","mixture-of-experts","moa","monte-carlo-tree-search","openai","openai-api","optimization","prompt-engineering","proxy-server"],"archived":false,"github_pushed_at":"2026-07-18T12:56:27+00:00","maintenance_label":"Steady","stars_delta_30d":67,"url":"https://www.graphcanon.com/tools/algorithmicsuperintelligence-optillm","markdown_url":"https://www.graphcanon.com/tools/algorithmicsuperintelligence-optillm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/algorithmicsuperintelligence-optillm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=algorithmicsuperintelligence-optillm","description":"Optimizing inference proxy for LLMs","homepage_url":null,"license":"Apache-2.0","open_issues":25,"watchers":40,"ai_summary":"A Python-based tool that optimizes the proxy servers for large language models (LLMs) and supports various deployment options including Docker.","readme_excerpt":"## 🚀 Quick Start\n\nGet powerful reasoning improvements in 3 simple steps:\n\n```bash\n\n---\n\n### Using docker\n\n```bash\ndocker pull ghcr.io/algorithmicsuperintelligence/optillm:latest\ndocker run -p 8000:8000 ghcr.io/algorithmicsuperintelligence/optillm:latest\n2024-10-22 07:45:05,612 - INFO - Loaded plugin: privacy\n2024-10-22 07:45:06,293 - INFO - Loaded plugin: memory\n2024-10-22 07:45:06,293 - INFO - Starting server with approach: auto\n```\n\n**Available Docker image variants:**\n\n- **Full image** (`latest`): Includes all dependencies for local inference and plugins\n- **Proxy-only** (`latest-proxy`): Lightweight image without local inference capabilities\n- **Offline** (`latest-offline`): Self-contained image with pre-downloaded models (spaCy) for fully offline operation\n\n```bash\n\n---\n\n### Install from source\n\nClone the repository with `git` and use `pip install` to setup the dependencies.\n\n```bash\ngit clone https://github.com/algorithmicsuperintelligence/optillm.git\ncd optillm\npython3 -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\n```\n\n---\n\n## Running with Docker\n\noptillm can optionally be built and run using Docker and the provided [Dockerfile](https://github.com/algorithmicsuperintelligence/optillm/blob/main/Dockerfile).\n\n---\n\n### Using Docker Compose\n\n1. Make sure you have Docker and Docker Compose installed on your system.\n\n2. Either update the environment variables in the docker-compose.yaml file or create a `.env` file in the project root directory and add any environment variables you want to set. For example, to set the OpenAI API key, add the following line to the `.env` file:\n\n   ```bash\n   OPENAI_API_KEY=your_openai_api_key_here\n   ```\n\n3. Run the following command to start optillm:\n\n   ```bash\n   docker compose up -d\n   ```\n\n   This will build the Docker image if it doesn't exist and start the optillm service.\n\n4. optillm will be available at `http://localhost:8000`.\n\nWhen using Docker, you can set these parameters as environment variables. For example, to set the approach and model, you would use:\n\n```bash\nOPTILLM_APPROACH=mcts\nOPTILLM_MODEL=gpt-4\n```\n\nTo secure the optillm proxy with an API key, set the `OPTILLM_API_KEY` environment variable:\n\n```bash\nOPTILLM_API_KEY=your_secret_api_key\n```\n\nWhen the API key is set, clients must include it in their requests using the `Authorization` header:\n\n```plain\nAuthorization: Bearer your_secret_api_key\n```","github_created_at":"2024-08-22T19:46:07+00:00","created_at":"2026-07-07T17:35:24.758444+00:00","updated_at":"2026-08-17T18:01:20.178641+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"agent","name":"agent"},{"slug":"agentic-ai","name":"agentic-ai"},{"slug":"genai","name":"genai"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"optimization","name":"optimization"},{"slug":"proxyservice","name":"proxyservice"}],"trust":{"provenance":{"is_fork":false,"github_id":846237240,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-17T18:01:17.819Z","maintenance":{"label":"Steady","score":60,"methodology":"github_public_v1","releases_90d":7,"days_since_push":30,"last_release_at":"2026-07-18T12:38:31Z","stars_delta_30d":67,"open_issues_delta_30d":5},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":1,"high_count":0,"last_scan_at":"2026-07-11T11:05:27.312Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-17T18:01:19.093Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-08-17T18:01:19.093Z","managed_saas":false},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-08-17T18:01:19.093Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-17T18:01:19.093Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-08-17T18:01:19.093Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-17T18:01:19.093Z"}},"decision_facts":{"hosting":{"model":"self_hosted","summary":"This open-source proxy supports diverse hosting environments and can be run via Docker for flexibility in deployment."},"pricing":{"model":"freemium","summary":"optillm is available under the Apache-2.0 license, which makes it free to use and distribute without cost."},"requirements":null,"constraints":{"hosting_model":"self_hosted","pricing_model":"freemium"},"when_to_use":["Use optillm when you require automatic optimization of the server approach to enhance reasoning capabilities with large language models.","Consider opting for optillm if your project involves agents or complex workflows managed via API gateway services.","If you need a Docker-based solution that can run without local inference capabilities, optillm's 'latest-proxy' image is appropriate."],"when_not_to_use":["Avoid optillm when your application does not require proxy server optimization for large language models; simpler serving setups may suffice.","Do not use optillm if your deployment environment strictly prohibits the use of Docker images or containers, given that this tool heavily relies on Docker for its various configurations."],"source":"enrich:decision_facts","observed_at":"2026-07-14T20:30:15.674Z"},"constraint_facets":{"hosting_model":"self_hosted","pricing_model":"freemium"},"decision_summary":[{"label":"Hosting","value":"self hosted - This open-source proxy supports diverse hosting environments and can be run via Docker for flexibility in deployment."},{"label":"Pricing","value":"freemium - optillm is available under the Apache-2.0 license, which makes it free to use and distribute without cost."},{"label":"Adopt for","value":"optillm is an optimizing inference proxy for LLMs that provides enhanced deployment options through Docker, supporting both full and lightweight configurations."}]}}