Home/Inference & Serving/vllm-mlx

vllm-mlx

waybarrios/vllm-mlx

Server for LLMs and vision-language models compatible with Apple Silicon

GitHub

GraphCanon updated 3w · GitHub synced 3w · 26 views this month

1.5k stars205 forksLast push 1mo Python Apache-2.0

Decision brief

vllm-mlx is an open-source inference server that runs large language models and vision-language models on Apple Silicon devices with continuous batching and multimodal support using native MLX backend.

Good fit when

If you need to run LLMs or vision-language models like Llama, Qwen-VL, and LLaVA efficiently on Apple Silicon devices.
When developing applications that require multimodal AI functionalities including text-to-speech, speech-to-text, and image-understanding, specifically on macOS systems.

Avoid when

If your target environment is not an Apple device equipped with the required hardware to run models via MLX backend.
When seeking a solution that offers high-speed token throughput beyond 400 tok/s as vllm-mlx may not be adequate for such performance needs.

Observed Jul 16, 2026 · Source: enrich:decision_facts

vllm-mlx alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Steady (31d since push); As of 3w
Provenance: Not a fork · Personal account; As of 3w
Security (OSV): No lockfile; As of 1mo

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install vllm-mlx

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

llmfit

Hundreds of models & providers. One command to find what runs on your hardware.

32k★

omlx

LLM inference server with continuous batching and SSD caching for Apple Silicon

19k★

open-llms

A list of open LLMs available for commercial use.

13k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

vllm-mlx is an open-source inference server that supports running large language models and vision-language models on Apple Silicon devices using native MLX backend.

Capability facts

CLI: CLI entrypoint
Source: pyproject.toml:[project.scripts] · Jul 30, 2026
Languages: python
Source: github.language+pyproject.toml · Jul 30, 2026

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Jul 30, 2026)

```python from openai import OpenAI

Source link

README

Quick start (30 seconds)

pip install vllm-mlx
vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching

OpenAI SDK:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
r = client.chat.completions.create(model="default", messages=[{"role": "user", "content": "Hi!"}])
print(r.choices[0].message.content)

Anthropic SDK / Claude Code:

export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=not-needed
claude

Installation

Using uv (recommended):

uv tool install vllm-mlx                 # CLI, system-wide

---

## License

Apache 2.0. See [LICENSE](LICENSE).

For agents

This page has a .md twin and JSON over the API.

/tools/waybarrios-vllm-mlx.md GET /api/graphcanon/tools/waybarrios-vllm-mlx GET /api/graphcanon/graph?tool=waybarrios-vllm-mlx