{"data":{"slug":"mosecorg-mosec","name":"mosec","tagline":"A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine","github_url":"https://github.com/mosecorg/mosec","owner":"mosecorg","repo":"mosec","owner_avatar_url":"https://avatars.githubusercontent.com/u/80561679?v=4","primary_language":"Python","stars":903,"forks":73,"topics":["cv","deep-learning","gpu","hacktoberfest","jax","llm","llm-serving","machine-learning","machine-learning-platform","mlops","model-serving","mxnet","nerual-network","python","pytorch","rust","tensorflow","tts"],"archived":false,"github_pushed_at":"2026-07-11T01:30:25+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/mosecorg-mosec","markdown_url":"https://www.graphcanon.com/tools/mosecorg-mosec.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/mosecorg-mosec","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=mosecorg-mosec","description":"A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine","homepage_url":"https://mosecorg.github.io/mosec/","license":"Apache-2.0","open_issues":17,"watchers":9,"ai_summary":null,"readme_excerpt":"## Installation\n\nMosec requires Python 3.7 or above. Install the latest [PyPI package](https://pypi.org/project/mosec/) for Linux or macOS with:\n\n```shell\npip install -U mosec\n\n---\n\n# or install with conda\nconda install conda-forge::mosec\n\n---\n\n# or install with pixi\npixi add mosec\n```\n\nTo build from the source code, install [Rust](https://www.rust-lang.org/) and run the following command:\n\n```shell\nmake package\n```\n\nYou will get a mosec wheel file in the `dist` folder.\n\n---\n\n## Deployment\n\n- If you're looking for a GPU base image with `mosec` installed, you can check the official image [`mosecorg/mosec`](https://hub.docker.com/r/mosecorg/mosec). For the complex use case, check out [envd](https://github.com/tensorchord/envd).\n- This service doesn't need Gunicorn or NGINX, but you can certainly use the ingress controller when necessary.\n- This service should be the PID 1 process in the container since it controls multiple processes. If you need to run multiple processes in one container, you will need a supervisor. You may choose [Supervisor](https://github.com/Supervisor/supervisor) or [Horust](https://github.com/FedericoPonzi/Horust).\n- Remember to collect the **metrics**.\n  - `mosec_service_batch_size_bucket` shows the batch size distribution.\n  - `mosec_service_batch_duration_second_bucket` shows the duration of dynamic batching for each connection in each stage (starts from receiving the first task).\n  - `mosec_service_process_duration_second_bucket` shows the duration of processing for each connection in each stage (including the IPC time but excluding the `mosec_service_batch_duration_second_bucket`).\n  - `mosec_service_remaining_task` shows the number of currently processing tasks.\n  - `mosec_service_throughput` shows the service throughput.\n- Stop the service with `SIGINT` (`CTRL+C`) or `SIGTERM` (`kill {PID}`) since it has the graceful shutdown logic.","github_created_at":"2021-03-13T04:07:20+00:00","created_at":"2026-07-11T23:12:15.501142+00:00","updated_at":"2026-07-11T23:12:28.247036+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"deep-learning","name":"deep-learning"},{"slug":"gpu","name":"gpu"},{"slug":"llm","name":"llm"},{"slug":"machine-learning","name":"machine-learning"},{"slug":"hacktoberfest","name":"hacktoberfest"},{"slug":"llm-serving","name":"llm-serving"},{"slug":"cv","name":"cv"},{"slug":"jax","name":"jax"}],"trust":{"provenance":{"is_fork":false,"github_id":347268387,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:12:18.320Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":1,"days_since_push":0,"last_release_at":"2026-04-15T14:34:28Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:12:18.783Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:12:18.073Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-07-11T23:12:18.073Z","managed_saas":false},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-07-11T23:12:18.073Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-11T23:12:18.073Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-07-11T23:12:18.073Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-11T23:12:18.073Z"}}}}