{"data":{"slug":"tensorchord-openmodelz","name":"openmodelz","tagline":"Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)","github_url":"https://github.com/tensorchord/openmodelz","owner":"tensorchord","repo":"openmodelz","owner_avatar_url":"https://avatars.githubusercontent.com/u/100543303?v=4","primary_language":"Go","stars":281,"forks":26,"topics":["cluster-manager","hacktoberfest","inference","llm","llmops","mlops"],"archived":false,"github_pushed_at":"2023-11-03T06:33:25+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/tensorchord-openmodelz","markdown_url":"https://www.graphcanon.com/tools/tensorchord-openmodelz.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/tensorchord-openmodelz","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=tensorchord-openmodelz","description":"Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)","homepage_url":"https://docs.open.modelz.ai","license":"Apache-2.0","open_issues":23,"watchers":7,"ai_summary":null,"readme_excerpt":"### Install `mdz`\n\nYou can install OpenModelZ using the following command:\n\n```text copy\npip install openmodelz\n```\n\nYou could verify the installation by running the following command:\n\n```text copy\nmdz\n```\n\nOnce you've installed the `mdz` you can start deploying models and experimenting with them.\n\n---\n\n### Create your first UI-based deployment\n\nOnce you've bootstrapped the `mdz` server, you can start deploying your first applications. We will use jupyter notebook as an example in this tutorial. You could use any docker image as your deployment.\n\n```text\n$ mdz deploy --image jupyter/minimal-notebook:lab-4.0.3 --name jupyter --port 8888 --command \"jupyter notebook --ip='*' --NotebookApp.token='' --NotebookApp.password=''\"\nInference jupyter is created\n$ mdz list\n NAME     ENDPOINT                                                   STATUS  INVOCATIONS  REPLICAS\n jupyter  http://jupyter-9pnxdkeb6jsfqkmq.192.168.71.93.modelz.live  Ready           488  1/1\n          http://192.168.71.93/inference/jupyter.default                                                                         \n```\n\nYou could access the deployment by visiting the endpoint URL. The endpoint will be automatically generated for each deployment with the following format: `<name>-<random-string>.<ip>.modelz.live`.\n\nIt is `http://jupyter-9pnxdkeb6jsfqkmq.192.168.71.93.modelz.live` in this case. The endpoint could be accessed from the outside world as well if you've provided the public IP address of your server to the `mdz server start` command.\n\n---\n\n### Scale your deployment\n\nYou could scale your deployment by using the `mdz scale` command.\n\n```text /scale/\n$ mdz scale simple-server --replicas 3\n```\n\nThe requests will be load balanced between the replicas of your deployment. \n\nYou could also tell the `mdz` to **autoscale your deployment** based on the inflight requests. Please check out the [Autoscaling](https://docs.open.modelz.ai/deployment/autoscale) documentation for more details.\n\n---\n\n### Debug your deployment\n\nSometimes you may want to debug your deployment. You could use the `mdz logs` command to get the logs of your deployment.\n\n```text /logs/\n$ mdz logs simple-server\nsimple-server-6756dd67ff-4bf4g: 10.42.0.1 - - [27/Jul/2023 02:32:16] \"GET / HTTP/1.1\" 200 -\nsimple-server-6756dd67ff-4bf4g: 10.42.0.1 - - [27/Jul/2023 02:32:16] \"GET / HTTP/1.1\" 200 -\nsimple-server-6756dd67ff-4bf4g: 10.42.0.1 - - [27/Jul/2023 02:32:17] \"GET / HTTP/1.1\" 200 -\n```\n\nYou could also use the `mdz exec` command to execute a command in the container of your deployment. You do not need to ssh into the server to do that.\n\n```text /exec/\n$ mdz exec simple-server ps\nPID   USER     TIME   COMMAND\n    1 root       0:00 /usr/bin/dumb-init /bin/sh -c python3 -m http.server 80\n    7 root       0:00 /bin/sh -c python3 -m http.server 80\n    8 root       0:00 python3 -m http.server 80\n    9 root       0:00 ps\n```\n\n```text /exec/\n$ mdz exec simple-server -ti bash\nbash-4.4# \n```\n\nOr you could port-forward the deployment to your local machine and debug it locally.\n\n```text /port-forward/\n$ mdz port-forward simple-server 7860\nForwarding inference simple-server to local port 7860\n```","github_created_at":"2023-07-13T14:50:20+00:00","created_at":"2026-07-11T23:29:24.076597+00:00","updated_at":"2026-07-11T23:29:37.79356+00:00","categories":[{"slug":"computer-vision","name":"Computer Vision","url":"https://www.graphcanon.com/categories/computer-vision","markdown_url":"https://www.graphcanon.com/categories/computer-vision.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/computer-vision"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"cluster-manager","name":"cluster-manager"},{"slug":"go","name":"go"},{"slug":"hacktoberfest","name":"hacktoberfest"},{"slug":"inference","name":"inference"},{"slug":"llm","name":"llm"},{"slug":"llmops","name":"llmops"},{"slug":"mlops","name":"mlops"}],"trust":{"provenance":{"is_fork":false,"github_id":666033021,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:29:33.054Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":981,"last_release_at":"2023-09-27T08:52:09Z"},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":106,"high_count":0,"last_scan_at":"2026-07-11T23:29:33.561Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:29:32.787Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-07-11T23:29:32.787Z"},"languages":{"value":["go","python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-11T23:29:32.787Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-11T23:29:32.787Z"}}}}