{"data":{"slug":"mosecorg-mosec","name":"mosec","tagline":"High-performance ML model serving framework","github_url":"https://github.com/mosecorg/mosec","owner":"mosecorg","repo":"mosec","owner_avatar_url":"https://avatars.githubusercontent.com/u/80561679?v=4","primary_language":"Python","stars":902,"forks":73,"topics":["cv","deep-learning","gpu","hacktoberfest","jax","llm","llm-serving","machine-learning","machine-learning-platform","mlops","model-serving","mxnet","nerual-network","python","pytorch","rust","tensorflow","tts"],"archived":false,"github_pushed_at":"2026-09-18T06:59:07+00:00","maintenance_label":"Very active","stars_delta_30d":-1,"url":"https://www.graphcanon.com/tools/mosecorg-mosec","markdown_url":"https://www.graphcanon.com/tools/mosecorg-mosec.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/mosecorg-mosec","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=mosecorg-mosec","description":"A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine","homepage_url":"https://mosecorg.github.io/mosec/","license":"Apache-2.0","open_issues":15,"watchers":9,"ai_summary":"Mosec is a framework for serving machine learning models with dynamic batching and support for CPU/GPU pipelines, enhancing compute machine exploitation.","readme_excerpt":"## Installation\n\nMosec requires Python 3.7 or above. Install the latest [PyPI package](https://pypi.org/project/mosec/) for Linux or macOS with:\n\n```shell\npip install -U mosec\n\n---\n\n# or install with conda\nconda install conda-forge::mosec\n\n---\n\n# or install with pixi\npixi add mosec\n```\n\nTo build from the source code, install [Rust](https://www.rust-lang.org/) and run the following command:\n\n```shell\nmake package\n```\n\nYou will get a mosec wheel file in the `dist` folder.\n\n---\n\n## Deployment\n\n- If you're looking for a GPU base image with `mosec` installed, you can check the official image [`mosecorg/mosec`](https://hub.docker.com/r/mosecorg/mosec). For the complex use case, check out [envd](https://github.com/tensorchord/envd).\n- This service doesn't need Gunicorn or NGINX, but you can certainly use the ingress controller when necessary.\n- This service should be the PID 1 process in the container since it controls multiple processes. If you need to run multiple processes in one container, you will need a supervisor. You may choose [Supervisor](https://github.com/Supervisor/supervisor) or [Horust](https://github.com/FedericoPonzi/Horust).\n- Remember to collect the **metrics**.\n  - `mosec_service_batch_size_bucket` shows the batch size distribution.\n  - `mosec_service_batch_duration_second_bucket` shows the duration of dynamic batching for each connection in each stage (starts from receiving the first task).\n  - `mosec_service_process_duration_second_bucket` shows the duration of processing for each connection in each stage (including the IPC time but excluding the `mosec_service_batch_duration_second_bucket`).\n  - `mosec_service_remaining_task` shows the number of currently processing tasks.\n  - `mosec_service_throughput` shows the service throughput.\n- Stop the service with `SIGINT` (`CTRL+C`) or `SIGTERM` (`kill {PID}`) since it has the graceful shutdown logic.","github_created_at":"2021-03-13T04:07:20+00:00","created_at":"2026-07-11T23:12:15.501142+00:00","updated_at":"2026-09-20T04:29:56.386891+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"cv","name":"cv"},{"slug":"deep-learning","name":"deep-learning"},{"slug":"gpu","name":"gpu"},{"slug":"jax","name":"jax"},{"slug":"llm","name":"llm"},{"slug":"llm-serving","name":"llm-serving"},{"slug":"machine-learning","name":"machine-learning"},{"slug":"mlops","name":"mlops"}],"trust":{"provenance":{"is_fork":false,"github_id":347268387,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:19:47.804Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":0,"last_release_at":"2026-04-15T14:34:28Z","stars_delta_30d":-1,"open_issues_delta_30d":-4},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:19:50.100Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:19:46.676Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-09-18T08:19:46.676Z","managed_saas":false},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-09-18T08:19:46.676Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-18T08:19:46.676Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-09-18T08:19:46.676Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:19:46.676Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Requires Python 3.7 or above.","Supports installation via PyPI, conda, or pixi.","Building from source requires Rust."],"min_ram_gb":null,"requires_docker":true},"constraints":{"min_ram_gb":null,"requires_docker":true},"when_to_use":["When you need to serve machine learning models with dynamic batching capabilities to optimize throughput.","If your application requires support for both CPU and GPU pipelines to maximize hardware utilization.","When deploying models in a containerized environment where the service should be the PID 1 process.","For scenarios where you want to avoid using Gunicorn or NGINX for serving your models."],"when_not_to_use":["If your deployment setup requires Gunicorn or NGINX for load balancing or reverse proxying.","When you do not need dynamic batching or have a simple model serving setup that does not benefit from CPU/GPU pipeline support.","If your deployment strategy does not involve containerization or you prefer not to have the service as the PID 1 process.","In cases where you are looking for a more traditional model serving solution that does not require Rust for building from source."],"source":"enrich:decision_facts","observed_at":"2026-09-18T10:15:39.052Z"},"constraint_facets":{"min_ram_gb":null,"requires_docker":true},"decision_summary":[{"label":"Requirements","value":"Requires Docker; Requires Python 3.7 or above.; Supports installation via PyPI, conda, or pixi.; Building from source requires Rust."},{"label":"Adopt for","value":"Mosec is a high-performance ML model serving framework that supports dynamic batching and CPU/GPU pipelines, making it suitable for environments where compute machine exploitation is critical."},{"label":"License detail","value":"Mosec is distributed under the Apache-2.0 license, allowing for free use, modification, and distribution."}]}}