{"data":{"slug":"mosecorg-mosec","name":"mosec","tagline":"A high-performance ML model serving framework with dynamic batching and CPU/GPU pipelines","github_url":"https://github.com/mosecorg/mosec","owner":"mosecorg","repo":"mosec","owner_avatar_url":"https://avatars.githubusercontent.com/u/80561679?v=4","primary_language":"Python","stars":903,"forks":73,"topics":["cv","deep-learning","gpu","hacktoberfest","jax","llm","llm-serving","machine-learning","machine-learning-platform","mlops","model-serving","mxnet","nerual-network","python","pytorch","rust","tensorflow","tts"],"archived":false,"github_pushed_at":"2026-08-01T02:19:14+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/mosecorg-mosec","markdown_url":"https://www.graphcanon.com/tools/mosecorg-mosec.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/mosecorg-mosec","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=mosecorg-mosec","description":"A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine","homepage_url":"https://mosecorg.github.io/mosec/","license":"Apache-2.0","open_issues":19,"watchers":9,"ai_summary":"Mosec is designed for efficient deployment of ML models, supporting various frameworks like JAX, PyTorch, MXNet, and TensorFlow.","readme_excerpt":"## Installation\n\nMosec requires Python 3.7 or above. Install the latest [PyPI package](https://pypi.org/project/mosec/) for Linux or macOS with:\n\n```shell\npip install -U mosec\n\n---\n\n# or install with conda\nconda install conda-forge::mosec\n\n---\n\n# or install with pixi\npixi add mosec\n```\n\nTo build from the source code, install [Rust](https://www.rust-lang.org/) and run the following command:\n\n```shell\nmake package\n```\n\nYou will get a mosec wheel file in the `dist` folder.\n\n---\n\n## Deployment\n\n- If you're looking for a GPU base image with `mosec` installed, you can check the official image [`mosecorg/mosec`](https://hub.docker.com/r/mosecorg/mosec). For the complex use case, check out [envd](https://github.com/tensorchord/envd).\n- This service doesn't need Gunicorn or NGINX, but you can certainly use the ingress controller when necessary.\n- This service should be the PID 1 process in the container since it controls multiple processes. If you need to run multiple processes in one container, you will need a supervisor. You may choose [Supervisor](https://github.com/Supervisor/supervisor) or [Horust](https://github.com/FedericoPonzi/Horust).\n- Remember to collect the **metrics**.\n  - `mosec_service_batch_size_bucket` shows the batch size distribution.\n  - `mosec_service_batch_duration_second_bucket` shows the duration of dynamic batching for each connection in each stage (starts from receiving the first task).\n  - `mosec_service_process_duration_second_bucket` shows the duration of processing for each connection in each stage (including the IPC time but excluding the `mosec_service_batch_duration_second_bucket`).\n  - `mosec_service_remaining_task` shows the number of currently processing tasks.\n  - `mosec_service_throughput` shows the service throughput.\n- Stop the service with `SIGINT` (`CTRL+C`) or `SIGTERM` (`kill {PID}`) since it has the graceful shutdown logic.","github_created_at":"2021-03-13T04:07:20+00:00","created_at":"2026-07-11T23:12:15.501142+00:00","updated_at":"2026-08-02T00:00:51.676709+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"cv","name":"cv"},{"slug":"deep-learning","name":"deep-learning"},{"slug":"gpu","name":"gpu"},{"slug":"jax","name":"jax"},{"slug":"llm","name":"llm"},{"slug":"llm-serving","name":"llm-serving"},{"slug":"machine-learning","name":"machine-learning"},{"slug":"machine-learning-platform","name":"machine-learning-platform"}],"trust":{"provenance":{"is_fork":false,"github_id":347268387,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-02T00:00:50.731Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":0,"last_release_at":"2026-04-15T14:34:28Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:12:18.783Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-02T00:00:51.232Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-08-02T00:00:51.232Z","managed_saas":false},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-08-02T00:00:51.232Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-02T00:00:51.232Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-08-02T00:00:51.232Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-02T00:00:51.232Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need dynamic batching to improve throughput on computational tasks","If your project prioritizes efficiency with both CPU and GPU pipelines","In cases where multiple machine learning frameworks such as JAX, PyTorch, MXNet, or TensorFlow are utilized"],"when_not_to_use":["Avoid if you require a tool that integrates directly with Gunicorn or NGINX for serving purposes","If your deployment environment relies on running more than one process in the container without a supervisor"],"source":"enrich:decision_facts","observed_at":"2026-07-14T18:38:51.728Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Mosec, Apache-2.0 licensed, is optimized for high-performance serving of ML models with dynamic batching and CPU/GPU support across different frameworks."}]}}