Paddler

Digital products and their users need privacy, reliability, cost control, and an option to be independent from closed-source model providers.

Paddler is an open-source LLM load balancer and serving platform. It allows you to run inference, deploy, and scale LLMs on your own infrastructure, providing a great developer experience along the way.

Key features

• Inference through a built-in llama.cpp engine
• LLM-specific load balancing
• Works through agents that can be added dynamically, allowing integration with autoscaling tools
• Request buffering, enabling scaling from zero hosts
• Dynamic model swapping
• Built-in web admin panel for management, monitoring, and testing
• Observability metrics

Who is Paddler for?

• Product teams that need LLM inference and embeddings in their features
• DevOps/LLMOps teams that need to run and deploy LLMs at scale
• Organizations handling sensitive data with high compliance and privacy requirements (medical, financial, etc.)
• Organizations wanting to achieve predictable LLM costs instead of being exposed to per-token pricing
• Product leaders who need reliable model performance to maintain a consistent user experience of their AI-based features

Community

• Discord https://discord.gg/92x3Z8a4gj
• Reddit (just started a subreddit, we will see how it goes :)) https://www.reddit.com/r/paddler/

Installation and Quickstart

Paddler is self-contained in a single binary file, so all you need to do to start using it is obtain the paddler binary and make it available in your system.

You can obtain the binary by:

• Option 1: Downloading the latest release from our GitHub releases
• Option 2: Or building Paddler from source (MSRV is 1.88.0)

Using Paddler

Once you have made the binary available in your system, you can start using Paddler. The entire Paddler functionality is available through the paddler command (running paddler --help will list all available commands).

There are only two deployable components, the balancer (which distributes the incoming requests), and the agent (which generates tokens and embeddings through slots).

To start the balancer, run:

paddler balancer --inference-addr 127.0.0.1:8061 --management-addr 127.0.0.1:8060 --web-admin-panel-addr 127.0.0.1:8062

The --web-admin-panel-addr flag is optional, but it will allow you to view your setup in a web browser.

And to start an agent with, for example, 4 slots, run:

paddler agent --management-addr 127.0.0.1:8060 --slots 4

Read more about the installation and setting up a basic cluster.

Documentation and resources

• Visit our documentation page to install Paddler and get started with it.
• API documentation is also available.
• Video overview
• FOSEDM 2026 talk - From Infrastructure to Production: A Year of Self-Hosted LLMs.

How does it work?

Paddler is built for an easy setup. It comes as a self-contained binary with only two deployable components, the balancer and the agents.

The balancer exposes the following:

• Inference service (used by applications that connect to it to obtain tokens or embeddings)
• Management service, which manages the Paddler's setup internally
• Web admin panel that lets you view and test your Paddler setup

Agents are usually deployed on separate instances. They further di