Requirements

• python 3.8 and above
• pytorch 1.12 and above, 2.0 and above are recommended
• transformers 4.32 and above
• CUDA 11.4 and above are recommended (this is for GPU users, flash-attention users, etc.)
  

pip install csrc/rotary

Now you can start with ModelScope or Transformers.

---

## 🐳 Docker

To simplify the deployment process, we provide docker images with pre-built environments: [qwenllm/qwen](https://hub.docker.com/r/qwenllm/qwen). You only need to install the driver and download model files to launch demos, deploy OpenAI API, and finetune the model.

---

# configure docker
sudo systemctl start docker

---

# test if docker is correctly installed
sudo docker run hello-world

---

### Deployment

Here we use Qwen-7B-Chat as an example. Before launching a web demo or API, you can setup the configuration as shown below:

```bash
IMAGE_NAME=qwenllm/qwen:cu117
PORT=8901
CHECKPOINT_PATH=/path/to/Qwen-7B-Chat   # Path to downloaded model checkpoints and codes

The following scripts can help you build:

• OpenAI API

bash docker/docker_openai_api.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH} --port ${PORT}

• Web UI

bash docker/docker_web_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH} --port ${PORT}

• CLI Demo

bash docker/docker_cli_demo.sh -i ${IMAGE_NAME} -c ${CHECKPOINT_PATH}

The commands above will automatically download the required image and launch a Web UI demo in background (the service will auto-restart). You can open http://localhost:${PORT} on the host to use the demo.

The demo is successfully launched if you see the following output:

Successfully started web demo. Open '...' to try!
Run `docker logs ...` to check demo status.
Run `docker rm -f ...` to stop and remove the demo.

If you want to check the status of the demo, you can use docker logs qwen to display outputs.

You can use docker rm -f qwen to stop the service and remove the container.

License Agreement

The source code provided at https://github.com/QwenLM/Qwen is licensed under the Apache 2.0 License that can be found at the root directory.

Researchers and developers are free to use the codes and model weights of both Qwen and Qwen-Chat. For their commercial use, please check the License Agreement accompanying each model.

• Qwen-72B, Qwen-14B, and Qwen-7B are licensed under the Tongyi Qianwen LICENSE AGREEMENT that can be found at the corresponding HuggingFace and ModelScope repository. For commercial use, please fill out the form (72B, 14B, and 7B) to apply.

• Qwen-1.8B is licensed under the Tongyi Qianwen RESEARCH LICENSE AGREEMENT that can be found at the corresponding HuggingFace and ModelScope repository. For commercial use, please contact us.