Getting Started with MII

DeepSpeed-MII allows users to create non-persistent and persistent deployments for supported models in just a few lines of code.

• Installation
• Non-Persistent Pipeline
• Persistent Deployment

---

Installation

The fasest way to get started is with our PyPI release of DeepSpeed-MII which means you can get started within minutes via:

pip install deepspeed-mii

For ease of use and significant reduction in lengthy compile times that many projects require in this space we distribute a pre-compiled python wheel covering the majority of our custom kernels through a new library called DeepSpeed-Kernels. We have found this library to be very portable across environments with NVIDIA GPUs with compute capabilities 8.0+ (Ampere+), CUDA 11.6+, and Ubuntu 20+. In most cases you shouldn't even need to know this library exists as it is a dependency of DeepSpeed-MII and will be installed with it. However, if for whatever reason you need to compile our kernels manually please see our advanced installation docs.

---

Persistent Deployment

A persistent deployment is ideal for use with long-running and production applications. The persistent model uses a lightweight GRPC server that can be queried by multiple clients at once. The full example for running a persistent model is only 5 lines. Give it a try!

import mii
client = mii.serve("mistralai/Mistral-7B-v0.1")
response = client.generate(["Deepspeed is", "Seattle is"], max_new_tokens=128)
print(response)

The returned response is a list of Response objects. We can access several details about the generation (e.g., response[0].prompt_length):

• generated_text: str Text generated by the model.
• prompt_length: int Number of tokens in the original prompt.
• generated_length: int Number of tokens generated.
• finish_reason: str Reason for stopping generation. stop indicates the EOS token was generated and length indicates the generation reached max_new_tokens or max_length.

If we want to generate text from other processes, we can do that too:

client = mii.client("mistralai/Mistral-7B-v0.1")
response = client.generate("Deepspeed is", max_new_tokens=128)

When we no longer need a persistent deployment, we can shutdown the server from any client:

client.terminate_server()

---

Persistent Deployment Options

While only the model name or path is required to stand up a persistent deployment, we offer customization options to our users.

mii.serve() Options:

• model_name_or_path: str (Required) Name or local path to a HuggingFace model.
• max_length: int (Defaults to maximum sequence length in model config) Sets the default maximum token length for the prompt + response.
• deployment_name: str (Defaults to f"{model_name_or_path}-mii-deployment") A unique identifying string for the persistent model. If provided, client objects should be retrieved with client = mii.client(deployment_name).
• tensor_parallel: int (Defaults to 1) Number of GPUs to split the model across.
• replica_num: int (Defaults to 1) The number of model replicas to stand up.
• enable_restful_api: bool (Defaults to False) When enabled, a RESTful API gateway process is launched that can be queried at http://{host}:{restful_api_port}/mii/{deployment_name}. See the section on RESTful APIs for more details.
• restful_api_port: int (Defaults to 28080) The port number used to interface with the RESTful API when enable_restful_api is set to True.

mii.client() Options:

• model_or_deployment_name: str Name of the model or deployment_name passed to mii.serve()

Users can also control the generation characteristic