Home/Inference & Serving/omlx

omlx

jundot/omlx

LLM inference server with continuous batching and SSD caching for Apple Silicon

GitHub Website

GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026

45views this month

22k stars1.9k forksLast push Sep 20, 2026 Python Apache-2.0

Decision brief

omlx is an LLM inference server tailored for Apple Silicon that emphasizes continuous batching and SSD caching capabilities, accessible through macOS menu bar control or Homebrew installation.

Good fit when

If your primary computing environment is based on Apple Silicon devices, omlx offers optimized performance for running large language model inferences.
When you require an inference server that supports continuous batching and leverages SSD caching for quicker service responses.

Avoid when

If your development infrastructure relies on non-Apple Silicon hardware, omlx's specific optimizations will not be as beneficial.
Teams that require cross-platform compatibility or run servers predominantly on non-macOS operating systems should consider alternatives with broader support.

Observed Jul 17, 2026 · Source: enrich:decision_facts

omlx alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Very active (0d since push); As of Sep 20, 2026
Provenance: Not a fork · Personal account; As of Sep 20, 2026
Security (OSV): No lockfile; As of Jul 15, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install omlx

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

inference

Unified production-ready inference API for various LLMs and models

9.6k★

whichllm

Command-line tool to find and benchmark local LLM performance

6.7k★

sie

Open-source inference server and production cluster for all the models your agent needs.

3.3k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

omlx is an LLM inference server designed to run on Apple Silicon that supports continuous batching and uses SSD caching. The tool can be managed via the macOS menu bar or through Homebrew installation.

Capability facts

CLI: CLI entrypoint
Source: pyproject.toml:[project.scripts] · Sep 20, 2026
Languages: python
Source: github.language+pyproject.toml · Sep 20, 2026

README

Managed background server (macOS app or Homebrew install) omlx start omlx stop omlx restart

For agents

This page has a .md twin and JSON over the API.

/tools/jundot-omlx.md GET /api/graphcanon/tools/jundot-omlx GET /api/graphcanon/graph?tool=jundot-omlx