mosec
A high-performance ML model serving framework with dynamic batching and CPU/GPU pipelines
GraphCanon updated 3w · GitHub synced 3w
Decision brief
Mosec, Apache-2.0 licensed, is optimized for high-performance serving of ML models with dynamic batching and CPU/GPU support across different frameworks.
Good fit when
- When you need dynamic batching to improve throughput on computational tasks
- If your project prioritizes efficiency with both CPU and GPU pipelines
Avoid when
- Avoid if you require a tool that integrates directly with Gunicorn or NGINX for serving purposes
- If your deployment environment relies on running more than one process in the container without a supervisor
Observed Jul 14, 2026 · Source: enrich:decision_facts
Verify the decision
Adoption
Package downloads where a registry match exists. GitHub stars (903) are secondary evidence.
- Docker Hub pulls (30d)
- 858·Docker Hub API·3w
Maintenance and security
Full trust report- Maintenance
- Very active (0d since push)
- As of 3w
- Provenance
- Not a fork · Organization account
- As of 3w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install mosecSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Mosec is designed for efficient deployment of ML models, supporting various frameworks like JAX, PyTorch, MXNet, and TensorFlow.
Capability facts
- Deploy
- Self-host
Source: dockerfile:Dockerfile · Aug 2, 2026
- Docker
- Dockerfile present
Source: dockerfile:Dockerfile · Aug 2, 2026
- CLI
- CLI entrypoint
Source: pyproject.toml:[project.scripts] · Aug 2, 2026
- Languages
- python
Source: github.language+pyproject.toml · Aug 2, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 2, 2026)
Mosec requires Python 3.7 or above. Install the latest [PyPI package](https://pypi.org/project/mosec/) foSource link
Tags
README
Installation
Mosec requires Python 3.7 or above. Install the latest PyPI package for Linux or macOS with:
pip install -U mosec
---
# or install with conda
conda install conda-forge::mosec
---
# or install with pixi
pixi add mosec
To build from the source code, install Rust and run the following command:
make package
You will get a mosec wheel file in the dist folder.
Deployment
- If you're looking for a GPU base image with
mosecinstalled, you can check the official imagemosecorg/mosec. For the complex use case, check out envd. - This service doesn't need Gunicorn or NGINX, but you can certainly use the ingress controller when necessary.
- This service should be the PID 1 process in the container since it controls multiple processes. If you need to run multiple processes in one container, you will need a supervisor. You may choose Supervisor or Horust.
- Remember to collect the metrics.
mosec_service_batch_size_bucketshows the batch size distribution.mosec_service_batch_duration_second_bucketshows the duration of dynamic batching for each connection in each stage (starts from receiving the first task).mosec_service_process_duration_second_bucketshows the duration of processing for each connection in each stage (including the IPC time but excluding themosec_service_batch_duration_second_bucket).mosec_service_remaining_taskshows the number of currently processing tasks.mosec_service_throughputshows the service throughput.
- Stop the service with
SIGINT(CTRL+C) orSIGTERM(kill {PID}) since it has the graceful shutdown logic.
For agents
This page has a .md twin and JSON over the API.