--- title: "paddler" type: "tool" slug: "intentee-paddler" canonical_url: "https://www.graphcanon.com/tools/intentee-paddler" github_url: "https://github.com/intentee/paddler" homepage_url: "https://paddler.intentee.com" stars: 1627 forks: 89 primary_language: "Rust" license: "Apache-2.0" categories: ["llm-frameworks", "inference-serving"] tags: ["llmops", "llm", "ai", "load-balancer", "llamacpp"] updated_at: "2026-07-07T18:46:26.154926+00:00" --- # paddler > LLM load balancer and serving platform for self-hosting LLMs/VLMs Paddler is a Rust-based, open-source LLM/VLM load balancer and serving platform designed to facilitate the deployment and scaling of large language models on private infrastructure. It features built-in llama.cpp inference engine, simple dynamic model swapping, request buffering, and an integrated web admin panel for management tasks. ## Facts - Repository: https://github.com/intentee/paddler - Homepage: https://paddler.intentee.com - Stars: 1,627 · Forks: 89 · Open issues: 25 · Watchers: 11 - Primary language: Rust - License: Apache-2.0 - Last pushed: 2026-07-07T08:14:17+00:00 ## Categories - [LLM Frameworks](/categories/llm-frameworks.md) - [Inference & Serving](/categories/inference-serving.md) ## Tags llmops, llm, ai, load-balancer, llamacpp ## Related tools - [ollama](/tools/ollama-ollama.md) - Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. (★ 175,659) - [prompts.chat](/tools/f-prompts-chat.md) - The world's largest open-source prompt library for AI (★ 165,019) - [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models (★ 162,347) - [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 144,575) - [awesome-llm-apps](/tools/shubhamsaboo-awesome-llm-apps.md) - 100+ AI Agent & RAG apps you can actually run — clone, customize, ship. (★ 116,702) - [LLMs-from-scratch](/tools/rasbt-llms-from-scratch.md) - Implement a ChatGPT-like LLM in PyTorch from scratch (★ 98,711) - [TradingAgents](/tools/tauricresearch-tradingagents.md) - TradingAgents: Multi-Agents LLM Financial Trading Framework (★ 91,610) - [caveman](/tools/juliusbrussee-caveman.md) - Cuts 65% of tokens in AI coding agent responses. (★ 86,150) ## README (excerpt) ```text # Paddler Digital products and their users need privacy, reliability, cost control, and an option to be independent from closed-source model providers. Paddler is an open-source LLM load balancer and serving platform. It allows you to run inference, deploy, and scale LLMs on your own infrastructure, providing a great developer experience along the way. ## Key features Paddler logo * Inference through a built-in [llama.cpp](https://github.com/ggml-org/llama.cpp) engine * LLM-specific load balancing * Works through agents that can be added dynamically, allowing integration with autoscaling tools * Request buffering, enabling scaling from zero hosts * Dynamic model swapping * Built-in web admin panel for management, monitoring, and testing * Observability metrics ## Who is Paddler for? * Product teams that need LLM inference and embeddings in their features * DevOps/LLMOps teams that need to run and deploy LLMs at scale * Organizations handling sensitive data with high compliance and privacy requirements (medical, financial, etc.) * Organizations wanting to achieve predictable LLM costs instead of being exposed to per-token pricing * Product leaders who need reliable model performance to maintain a consistent user experience of their AI-based features ## Community - Discord https://discord.gg/92x3Z8a4gj - Reddit (just started a subreddit, we will see how it goes :)) https://www.reddit.com/r/paddler/ ## Installation and Quickstart Paddler is self-contained in a single binary file, so all you need to do to start using it is obtain the `paddler` binary and make it available in your system. You can obtain the binary by: * Option 1: Downloading the latest release from our [GitHub releases](https://github.com/intentee/paddler/releases) * Option 2: Or building Paddler from source (MSRV is *1.88.0*) ### Using Paddler Once you have made the binary available in your system, you can start using Paddler. The entire Paddler functionality is available through the `paddler` command (running `paddler --help` will list all available commands). There are only two deployable components, the `balancer` (which distributes the incoming requests), and the `agent` (which generates tokens and embeddings through slots). To start the balancer, run: ```sh paddler balancer --inference-addr 127.0.0.1:8061 --management-addr 127.0.0.1:8060 --web-admin-panel-addr 127.0.0.1:8062 ``` The `--web-admin-panel-addr` flag is optional, but it will allow you to view your setup in a web browser. And to start an agent with, for example, 4 slots, run: ```sh paddler agent --management-addr 127.0.0.1:8060 --slots 4 ``` Read more about the [installation](https://paddler.intentee.com/docs/introduction/installation/) and [setting up a basic cluster](https://paddler.intentee.com/docs/starting-out/set-up-a-basic-llm-cluster/). ## Documentation and resources - Visit our [documentation page](https://paddler.intentee.com/docs/introduction/what-is-paddler/) to install Paddler and get started with it. - [API documentation](https://paddler.intentee.com/api/introduction/using-paddler-api/) is also available. - [Video overview](https://www.youtube.com/watch?v=aT6QCL8lk08) - [FOSEDM 2026 talk](https://fosdem.org/2026/schedule/event/PD8WGF-from_infrastructure_to_production_a_year_of_self-hosted_llms/) - From Infrastructure to Production: A Year of Self-Hosted LLMs. ## How does it work? Paddler is built for an easy setup. It comes as a self-contained binary with only two deployable components, the `balancer` and the `agents`. The `balancer` exposes the following: - Inference service (used by applications that connect to it to obtain tokens or embeddings) - Management service, which manages the Paddler's setup internally - Web admin panel that lets you view and test your Paddler setup `Agents` are usually deployed on separate instances. They further di ``` --- **Machine-readable endpoints** - JSON: [`/api/graphcanon/tools/intentee-paddler`](/api/graphcanon/tools/intentee-paddler) - LLM index: [/llms.txt](/llms.txt) - Full corpus: [/llms-full.txt](/llms-full.txt) _GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_