{"data":{"slug":"deepspeedai-deepspeed-mii","name":"DeepSpeed-MII","tagline":"Enables low-latency and high-throughput inference for deep learning models","github_url":"https://github.com/deepspeedai/DeepSpeed-MII","owner":"deepspeedai","repo":"DeepSpeed-MII","owner_avatar_url":"https://avatars.githubusercontent.com/u/74068820?v=4","primary_language":"Python","stars":2113,"forks":192,"topics":["deep-learning","inference","pytorch"],"archived":false,"github_pushed_at":"2025-06-30T16:21:45+00:00","maintenance_label":"Dormant","stars_delta_30d":5,"url":"https://www.graphcanon.com/tools/deepspeedai-deepspeed-mii","markdown_url":"https://www.graphcanon.com/tools/deepspeedai-deepspeed-mii.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/deepspeedai-deepspeed-mii","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=deepspeedai-deepspeed-mii","description":"MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.","homepage_url":null,"license":"Apache-2.0","open_issues":209,"watchers":41,"ai_summary":"DeepSpeed-MII is a tool that allows for the creation of non-persistent and persistent deployments for supported deep learning models, focusing on inference with low latency and high throughput. It supports PyTorch and is distributed with pre-compiled Python wheels for ease of use.","readme_excerpt":"# Getting Started with MII\n\nDeepSpeed-MII allows users to create non-persistent and persistent deployments for supported models in just a few lines of code.\n\n- [Installation](#installation)\n- [Non-Persistent Pipeline](#non-persistent-pipeline)\n- [Persistent Deployment](#persistent-deployment)\n\n---\n\n## Installation\n\nThe fasest way to get started is with our [PyPI release of DeepSpeed-MII](https://pypi.org/project/deepspeed-mii/) which means you can get started within minutes via:\n\n```bash\npip install deepspeed-mii\n```\n\nFor ease of use and significant reduction in lengthy compile times that many projects require in this space we distribute a pre-compiled python wheel covering the majority of our custom kernels through a new library called [DeepSpeed-Kernels](https://github.com/deepspeedai/DeepSpeed-Kernels). We have found this library to be very portable across environments with NVIDIA GPUs with compute capabilities 8.0+ (Ampere+), CUDA 11.6+, and Ubuntu 20+. In most cases you shouldn't even need to know this library exists as it is a dependency of DeepSpeed-MII and will be installed with it. However, if for whatever reason you need to compile our kernels manually please see our [advanced installation docs](https://github.com/deepspeedai/DeepSpeed-Kernels#source).\n\n---\n\n## Persistent Deployment\n\nA persistent deployment is ideal for use with long-running and production applications. The persistent model uses a lightweight GRPC server that can be queried by multiple clients at once. The full example for running a persistent model is only 5 lines. Give it a try!\n\n```python\nimport mii\nclient = mii.serve(\"mistralai/Mistral-7B-v0.1\")\nresponse = client.generate([\"Deepspeed is\", \"Seattle is\"], max_new_tokens=128)\nprint(response)\n```\n\nThe returned `response` is a list of `Response` objects. We can access several details about the generation (e.g., `response[0].prompt_length`):\n\n- `generated_text: str` Text generated by the model.\n- `prompt_length: int` Number of tokens in the original prompt.\n- `generated_length: int` Number of tokens generated.\n- `finish_reason: str` Reason for stopping generation. `stop` indicates the EOS token was generated and `length` indicates the generation reached `max_new_tokens` or `max_length`.\n\nIf we want to generate text from other processes, we can do that too:\n\n```python\nclient = mii.client(\"mistralai/Mistral-7B-v0.1\")\nresponse = client.generate(\"Deepspeed is\", max_new_tokens=128)\n```\n\nWhen we no longer need a persistent deployment, we can shutdown the server from any client:\n\n```python\nclient.terminate_server()\n```\n\n---\n\n### Persistent Deployment Options\nWhile only the model name or path is required to stand up a persistent deployment, we offer customization options to our users.\n\n**`mii.serve()` Options**:\n- `model_name_or_path: str` (Required) Name or local path to a [HuggingFace](https://huggingface.co/) model.\n- `max_length: int` (Defaults to maximum sequence length in model config) Sets the default maximum token length for the prompt + response.\n- `deployment_name: str` (Defaults to `f\"{model_name_or_path}-mii-deployment\"`) A unique identifying string for the persistent model. If provided, client objects should be retrieved with `client = mii.client(deployment_name)`.\n- `tensor_parallel: int` (Defaults to `1`) Number of GPUs to split the model across.\n- `replica_num: int` (Defaults to `1`) The number of model replicas to stand up.\n- `enable_restful_api: bool` (Defaults to `False`) When enabled, a RESTful API gateway process is launched that can be queried at `http://{host}:{restful_api_port}/mii/{deployment_name}`. See the [section on RESTful APIs](#restful-api) for more details.\n- `restful_api_port: int` (Defaults to `28080`) The port number used to interface with the RESTful API when `enable_restful_api` is set to `True`.\n\n**`mii.client()` Options**:\n- `model_or_deployment_name: str` Name of the model or `deployment_name` passed to `mii.serve()`\n\nUsers can also control the generation characteristic","github_created_at":"2022-03-23T22:30:45+00:00","created_at":"2026-07-11T10:37:39.630572+00:00","updated_at":"2026-09-20T04:29:35.872847+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"deep-learning","name":"deep-learning"},{"slug":"grpc-server","name":"grpc-server"},{"slug":"high-throughput","name":"high-throughput"},{"slug":"inference","name":"inference"},{"slug":"low-latency","name":"low-latency"},{"slug":"non-persistent-pipeline","name":"non-persistent-pipeline"},{"slug":"persistent-deployment","name":"persistent-deployment"},{"slug":"pytorch","name":"pytorch"}],"trust":{"provenance":{"is_fork":false,"github_id":473378688,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:17:02.870Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":444,"last_release_at":"2025-03-25T19:50:02Z","stars_delta_30d":5,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:17:05.046Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:17:01.781Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-18T08:17:01.781Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:17:01.781Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium","summary":"DeepSpeed-MII is open-source under the Apache-2.0 license, making it free to use, but users may incur costs based on the infrastructure they use to run the models."},"requirements":{"notes":["Requires NVIDIA GPUs with compute capabilities 8.0+ (Ampere+), CUDA 11.6+, and Ubuntu 20+."],"min_ram_gb":8,"requires_docker":false},"constraints":{"min_ram_gb":8,"pricing_model":"freemium","requires_docker":false},"when_to_use":["Use DeepSpeed-MII when you need to deploy deep learning models with low latency and high throughput, especially for production applications that require persistent deployment.","Choose DeepSpeed-MII if you are working with PyTorch models and want to avoid lengthy compile times, as it comes with pre-compiled Python wheels for easy setup."],"when_not_to_use":["Avoid DeepSpeed-MII if your environment does not meet the requirements of NVIDIA GPUs with compute capabilities 8.0+ (Ampere+), CUDA 11.6+, and Ubuntu 20+.","Do not use DeepSpeed-MII if you require support for frameworks other than PyTorch, as it is specifically designed for PyTorch models."],"source":"enrich:decision_facts","observed_at":"2026-09-18T10:02:45.161Z"},"constraint_facets":{"min_ram_gb":8,"pricing_model":"freemium","requires_docker":false},"decision_summary":[{"label":"Pricing","value":"freemium - DeepSpeed-MII is open-source under the Apache-2.0 license, making it free to use, but users may incur costs based on the infrastructure they use to run the models."},{"label":"Requirements","value":"Min 8 GB RAM; Requires NVIDIA GPUs with compute capabilities 8.0+ (Ampere+), CUDA 11.6+, and Ubuntu 20+."},{"label":"Adopt for","value":"DeepSpeed-MII is a tool for low-latency and high-throughput inference in deep learning models, supporting PyTorch and offering both non-persistent and persistent deployment options."}]}}