{"data":{"slug":"tensorchord-openmodelz","name":"openmodelz","tagline":"Automate and scale inference of large language models on Kubernetes.","github_url":"https://github.com/tensorchord/openmodelz","owner":"tensorchord","repo":"openmodelz","owner_avatar_url":"https://avatars.githubusercontent.com/u/100543303?v=4","primary_language":"Go","stars":284,"forks":26,"topics":["cluster-manager","hacktoberfest","inference","llm","llmops","mlops"],"archived":false,"github_pushed_at":"2023-11-03T06:33:25+00:00","maintenance_label":"Dormant","stars_delta_30d":2,"url":"https://www.graphcanon.com/tools/tensorchord-openmodelz","markdown_url":"https://www.graphcanon.com/tools/tensorchord-openmodelz.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/tensorchord-openmodelz","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=tensorchord-openmodelz","description":"Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)","homepage_url":"https://docs.open.modelz.ai","license":"Apache-2.0","open_issues":23,"watchers":7,"ai_summary":"OpenModelZ is a tool designed to ease the deployment, scaling, and management of large language model inferences on clusters managed by Kubernetes or other systems. It supports automatic scaling of deployments based on current load and provides features for deploying custom applications.","readme_excerpt":"### Install `mdz`\n\nYou can install OpenModelZ using the following command:\n\n```text copy\npip install openmodelz\n```\n\nYou could verify the installation by running the following command:\n\n```text copy\nmdz\n```\n\nOnce you've installed the `mdz` you can start deploying models and experimenting with them.\n\n---\n\n### Create your first UI-based deployment\n\nOnce you've bootstrapped the `mdz` server, you can start deploying your first applications. We will use jupyter notebook as an example in this tutorial. You could use any docker image as your deployment.\n\n```text\n$ mdz deploy --image jupyter/minimal-notebook:lab-4.0.3 --name jupyter --port 8888 --command \"jupyter notebook --ip='*' --NotebookApp.token='' --NotebookApp.password=''\"\nInference jupyter is created\n$ mdz list\n NAME     ENDPOINT                                                   STATUS  INVOCATIONS  REPLICAS\n jupyter  http://jupyter-9pnxdkeb6jsfqkmq.192.168.71.93.modelz.live  Ready           488  1/1\n          http://192.168.71.93/inference/jupyter.default                                                                         \n```\n\nYou could access the deployment by visiting the endpoint URL. The endpoint will be automatically generated for each deployment with the following format: `<name>-<random-string>.<ip>.modelz.live`.\n\nIt is `http://jupyter-9pnxdkeb6jsfqkmq.192.168.71.93.modelz.live` in this case. The endpoint could be accessed from the outside world as well if you've provided the public IP address of your server to the `mdz server start` command.\n\n---\n\n### Scale your deployment\n\nYou could scale your deployment by using the `mdz scale` command.\n\n```text /scale/\n$ mdz scale simple-server --replicas 3\n```\n\nThe requests will be load balanced between the replicas of your deployment. \n\nYou could also tell the `mdz` to **autoscale your deployment** based on the inflight requests. Please check out the [Autoscaling](https://docs.open.modelz.ai/deployment/autoscale) documentation for more details.\n\n---\n\n### Debug your deployment\n\nSometimes you may want to debug your deployment. You could use the `mdz logs` command to get the logs of your deployment.\n\n```text /logs/\n$ mdz logs simple-server\nsimple-server-6756dd67ff-4bf4g: 10.42.0.1 - - [27/Jul/2023 02:32:16] \"GET / HTTP/1.1\" 200 -\nsimple-server-6756dd67ff-4bf4g: 10.42.0.1 - - [27/Jul/2023 02:32:16] \"GET / HTTP/1.1\" 200 -\nsimple-server-6756dd67ff-4bf4g: 10.42.0.1 - - [27/Jul/2023 02:32:17] \"GET / HTTP/1.1\" 200 -\n```\n\nYou could also use the `mdz exec` command to execute a command in the container of your deployment. You do not need to ssh into the server to do that.\n\n```text /exec/\n$ mdz exec simple-server ps\nPID   USER     TIME   COMMAND\n    1 root       0:00 /usr/bin/dumb-init /bin/sh -c python3 -m http.server 80\n    7 root       0:00 /bin/sh -c python3 -m http.server 80\n    8 root       0:00 python3 -m http.server 80\n    9 root       0:00 ps\n```\n\n```text /exec/\n$ mdz exec simple-server -ti bash\nbash-4.4# \n```\n\nOr you could port-forward the deployment to your local machine and debug it locally.\n\n```text /port-forward/\n$ mdz port-forward simple-server 7860\nForwarding inference simple-server to local port 7860\n```","github_created_at":"2023-07-13T14:50:20+00:00","created_at":"2026-07-11T23:29:24.076597+00:00","updated_at":"2026-09-29T06:00:15.703168+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"cluster-manager","name":"cluster-manager"},{"slug":"hacktoberfest","name":"hacktoberfest"},{"slug":"inference","name":"inference"},{"slug":"llm","name":"llm"},{"slug":"llmops","name":"llmops"},{"slug":"mlops","name":"mlops"}],"trust":{"provenance":{"is_fork":false,"github_id":666033021,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-29T06:00:14.852Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1060,"last_release_at":"2023-09-27T08:52:09Z","stars_delta_30d":2,"open_issues_delta_30d":0},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":106,"high_count":0,"last_scan_at":"2026-07-11T23:29:33.561Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-29T06:00:15.377Z"},"has_cli":{"value":true,"source":"pyproject.toml:[project.scripts]","observed_at":"2026-09-29T06:00:15.377Z"},"languages":{"value":["go","python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-29T06:00:15.377Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-29T06:00:15.377Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need automatic scaling of large language models based on current load on Kubernetes clusters.","If you require deploying custom applications with an emphasis on ease-of-use for managing LLM inferences."],"when_not_to_use":["Avoid using if your deployment setup does not include Kubernetes or another cluster management system that OpenModelZ supports.","Do not use this tool if you do not need automatic scaling features, as manual setup might be more straightforward for simpler deployments."],"source":"enrich:decision_facts","observed_at":"2026-07-16T22:03:36.858Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"OpenModelZ automates and scales large language model inferences on Kubernetes."}]}}