openmodelz
Automate and scale inference of large language models on Kubernetes.
GraphCanon updated 3w · GitHub synced 3w
Decision brief
OpenModelZ automates and scales large language model inferences on Kubernetes.
Good fit when
- When you need automatic scaling of large language models based on current load on Kubernetes clusters.
- If you require deploying custom applications with an emphasis on ease-of-use for managing LLM inferences.
Avoid when
- Avoid using if your deployment setup does not include Kubernetes or another cluster management system that OpenModelZ supports.
- Do not use this tool if you do not need automatic scaling features, as manual setup might be more straightforward for simpler deployments.
Observed Jul 16, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (1004d since push)
- As of 3w
- Provenance
- Not a fork · Organization account
- As of 3w
- Security (OSV)
- 106 low (106 low)
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Backing
Company context for TensorChord. Display-only - separate from trust and ranking.
- Company
- TensorChord·GitHub org profile·1mo
- Commercial model
- Pure OSS·GitHub org profile (public repos)·1mo
Install
go get github.com/tensorchord/openmodelzSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
OpenModelZ is a tool designed to ease the deployment, scaling, and management of large language model inferences on clusters managed by Kubernetes or other systems. It supports automatic scaling of deployments based on current load and provides features for deploying custom applications.
Capability facts
- CLI
- CLI entrypoint
Source: pyproject.toml:[project.scripts] · Aug 3, 2026
- Languages
- go, python
Source: github.language+pyproject.toml · Aug 3, 2026
Categories
Graph entities
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 3, 2026)
1 root 0:00 /usr/bin/dumb-init /bin/sh -c python3 -m http.server 80Source link
Tags
README
Install mdz
You can install OpenModelZ using the following command:
pip install openmodelz
You could verify the installation by running the following command:
mdz
Once you've installed the mdz you can start deploying models and experimenting with them.
Create your first UI-based deployment
Once you've bootstrapped the mdz server, you can start deploying your first applications. We will use jupyter notebook as an example in this tutorial. You could use any docker image as your deployment.
$ mdz deploy --image jupyter/minimal-notebook:lab-4.0.3 --name jupyter --port 8888 --command "jupyter notebook --ip='*' --NotebookApp.token='' --NotebookApp.password=''"
Inference jupyter is created
$ mdz list
NAME ENDPOINT STATUS INVOCATIONS REPLICAS
jupyter http://jupyter-9pnxdkeb6jsfqkmq.192.168.71.93.modelz.live Ready 488 1/1
http://192.168.71.93/inference/jupyter.default
You could access the deployment by visiting the endpoint URL. The endpoint will be automatically generated for each deployment with the following format: <name>-<random-string>.<ip>.modelz.live.
It is http://jupyter-9pnxdkeb6jsfqkmq.192.168.71.93.modelz.live in this case. The endpoint could be accessed from the outside world as well if you've provided the public IP address of your server to the mdz server start command.
Scale your deployment
You could scale your deployment by using the mdz scale command.
$ mdz scale simple-server --replicas 3
The requests will be load balanced between the replicas of your deployment.
You could also tell the mdz to autoscale your deployment based on the inflight requests. Please check out the Autoscaling documentation for more details.
Debug your deployment
Sometimes you may want to debug your deployment. You could use the mdz logs command to get the logs of your deployment.
$ mdz logs simple-server
simple-server-6756dd67ff-4bf4g: 10.42.0.1 - - [27/Jul/2023 02:32:16] "GET / HTTP/1.1" 200 -
simple-server-6756dd67ff-4bf4g: 10.42.0.1 - - [27/Jul/2023 02:32:16] "GET / HTTP/1.1" 200 -
simple-server-6756dd67ff-4bf4g: 10.42.0.1 - - [27/Jul/2023 02:32:17] "GET / HTTP/1.1" 200 -
You could also use the mdz exec command to execute a command in the container of your deployment. You do not need to ssh into the server to do that.
$ mdz exec simple-server ps
PID USER TIME COMMAND
1 root 0:00 /usr/bin/dumb-init /bin/sh -c python3 -m http.server 80
7 root 0:00 /bin/sh -c python3 -m http.server 80
8 root 0:00 python3 -m http.server 80
9 root 0:00 ps
$ mdz exec simple-server -ti bash
bash-4.4#
Or you could port-forward the deployment to your local machine and debug it locally.
$ mdz port-forward simple-server 7860
Forwarding inference simple-server to local port 7860
For agents
This page has a .md twin and JSON over the API.