{"data":{"slug":"deepspeedai-deepspeed-mii","name":"DeepSpeed-MII","tagline":"Low-latency and high-throughput inference for deep learning models","github_url":"https://github.com/deepspeedai/DeepSpeed-MII","owner":"deepspeedai","repo":"DeepSpeed-MII","owner_avatar_url":"https://avatars.githubusercontent.com/u/74068820?v=4","primary_language":"Python","stars":2109,"forks":191,"topics":["deep-learning","inference","pytorch"],"archived":false,"github_pushed_at":"2025-06-30T16:21:45+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/deepspeedai-deepspeed-mii","markdown_url":"https://www.graphcanon.com/tools/deepspeedai-deepspeed-mii.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/deepspeedai-deepspeed-mii","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=deepspeedai-deepspeed-mii","description":"MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.","homepage_url":null,"license":"Apache-2.0","open_issues":209,"watchers":41,"ai_summary":"DeepSpeed-MII enables users to deploy non-persistent and persistent inference services for supported AI models efficiently by minimizing setup and compile times through pre-compiled Python wheels.","readme_excerpt":"# Getting Started with MII\n\nDeepSpeed-MII allows users to create non-persistent and persistent deployments for supported models in just a few lines of code.\n\n- [Installation](#installation)\n- [Non-Persistent Pipeline](#non-persistent-pipeline)\n- [Persistent Deployment](#persistent-deployment)\n\n---\n\n## Installation\n\nThe fasest way to get started is with our [PyPI release of DeepSpeed-MII](https://pypi.org/project/deepspeed-mii/) which means you can get started within minutes via:\n\n```bash\npip install deepspeed-mii\n```\n\nFor ease of use and significant reduction in lengthy compile times that many projects require in this space we distribute a pre-compiled python wheel covering the majority of our custom kernels through a new library called [DeepSpeed-Kernels](https://github.com/deepspeedai/DeepSpeed-Kernels). We have found this library to be very portable across environments with NVIDIA GPUs with compute capabilities 8.0+ (Ampere+), CUDA 11.6+, and Ubuntu 20+. In most cases you shouldn't even need to know this library exists as it is a dependency of DeepSpeed-MII and will be installed with it. However, if for whatever reason you need to compile our kernels manually please see our [advanced installation docs](https://github.com/deepspeedai/DeepSpeed-Kernels#source).\n\n---\n\n## Persistent Deployment\n\nA persistent deployment is ideal for use with long-running and production applications. The persistent model uses a lightweight GRPC server that can be queried by multiple clients at once. The full example for running a persistent model is only 5 lines. Give it a try!\n\n```python\nimport mii\nclient = mii.serve(\"mistralai/Mistral-7B-v0.1\")\nresponse = client.generate([\"Deepspeed is\", \"Seattle is\"], max_new_tokens=128)\nprint(response)\n```\n\nThe returned `response` is a list of `Response` objects. We can access several details about the generation (e.g., `response[0].prompt_length`):\n\n- `generated_text: str` Text generated by the model.\n- `prompt_length: int` Number of tokens in the original prompt.\n- `generated_length: int` Number of tokens generated.\n- `finish_reason: str` Reason for stopping generation. `stop` indicates the EOS token was generated and `length` indicates the generation reached `max_new_tokens` or `max_length`.\n\nIf we want to generate text from other processes, we can do that too:\n\n```python\nclient = mii.client(\"mistralai/Mistral-7B-v0.1\")\nresponse = client.generate(\"Deepspeed is\", max_new_tokens=128)\n```\n\nWhen we no longer need a persistent deployment, we can shutdown the server from any client:\n\n```python\nclient.terminate_server()\n```\n\n---\n\n### Persistent Deployment Options\nWhile only the model name or path is required to stand up a persistent deployment, we offer customization options to our users.\n\n**`mii.serve()` Options**:\n- `model_name_or_path: str` (Required) Name or local path to a [HuggingFace](https://huggingface.co/) model.\n- `max_length: int` (Defaults to maximum sequence length in model config) Sets the default maximum token length for the prompt + response.\n- `deployment_name: str` (Defaults to `f\"{model_name_or_path}-mii-deployment\"`) A unique identifying string for the persistent model. If provided, client objects should be retrieved with `client = mii.client(deployment_name)`.\n- `tensor_parallel: int` (Defaults to `1`) Number of GPUs to split the model across.\n- `replica_num: int` (Defaults to `1`) The number of model replicas to stand up.\n- `enable_restful_api: bool` (Defaults to `False`) When enabled, a RESTful API gateway process is launched that can be queried at `http://{host}:{restful_api_port}/mii/{deployment_name}`. See the [section on RESTful APIs](#restful-api) for more details.\n- `restful_api_port: int` (Defaults to `28080`) The port number used to interface with the RESTful API when `enable_restful_api` is set to `True`.\n\n**`mii.client()` Options**:\n- `model_or_deployment_name: str` Name of the model or `deployment_name` passed to `mii.serve()`\n\nUsers can also control the generation characteristic","github_created_at":"2022-03-23T22:30:45+00:00","created_at":"2026-07-11T10:37:39.630572+00:00","updated_at":"2026-07-11T17:57:56.166567+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"deep-learning","name":"deep-learning"},{"slug":"pytorch","name":"pytorch"},{"slug":"inference","name":"inference"}],"trust":{"provenance":{"is_fork":false,"github_id":473378688,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:37:40.253Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":375,"last_release_at":"2025-03-25T19:50:02Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:37:41.118Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T17:57:55.530Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-11T17:57:55.530Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-11T17:57:55.530Z"}}}}