Home/Inference & Serving/LLMKube

LLMKube

defilantech/LLMKube

Kubernetes operator for self-hosted LLM inference across heterogeneous GPU fleets

GitHub Website

GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026

39views this month

210 stars33 forksLast push Sep 17, 2026 Go Apache-2.0

Decision brief

LLMKube is a Kubernetes operator for deploying and managing self-hosted large language model inference services on various GPU architectures, including NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. It supports runtim

Good fit when

When you need to deploy and manage self-hosted large language model inference services across a variety of GPU architectures, including NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal.
If you require support for multiple runtimes like llama.cpp, vLLM, TGI, and mlx-server, and features such as multi-GPU sharding, model caching, and OpenAI-compatible endpoints.

Avoid when

If your infrastructure does not include a Kubernetes cluster, as LLMKube is designed to operate within a Kubernetes environment.
If you are looking for a solution that does not support a heterogeneous mix of GPU architectures, LLMKube is specifically designed to work with NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal.

Pricing:: freemium - Free and open-source under the Apache-2.0 license.
Requirements:: Requires a Kubernetes cluster for installation and operation.; Supports deployment on a variety of GPU architectures, including NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal.

Observed Sep 18, 2026 · Source: enrich:decision_facts

LLMKube alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Very active (0d since push); As of Sep 18, 2026
Provenance: Not a fork · Organization account; As of Sep 18, 2026
Security (OSV): 3 low (3 low); As of Sep 18, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

go get github.com/defilantech/LLMKube

pkg.go.dev

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

92k★

omlx

LLM inference server with continuous batching and SSD caching for Apple Silicon

22k★

TensorRT-LLM

Python API for defining and optimizing Large Language Models for efficient inference on NVIDIA GPUs

15k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

LLMKube is a Kubernetes operator designed for deploying and managing self-hosted large language model inference services on a variety of GPU architectures, including NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. It supports multiple runtimes like llama.cpp, vLLM, TGI, and mlx-server, and offers features such as multi-GPU sharding, model caching, and OpenAI-compatible endpoints.

Capability facts

Deploy: Self-host
Source: dockerfile:Dockerfile · Sep 18, 2026
Docker: Dockerfile present
Source: dockerfile:Dockerfile · Sep 18, 2026
Languages: go
Source: github.language · Sep 18, 2026

README

Install the CLI brew install defilantech/tap/llmkube Install the operator on any K8s cluster helm repo add llmkube https://defilantech.github.io/LLMKube helm install llmkube llmkube/llmkube namespace llmkube system create namespace License Apache 2.0 — see LICENSE.

For agents

This page has a .md twin and JSON over the API.

/tools/defilantech-llmkube.md GET /api/graphcanon/tools/defilantech-llmkube GET /api/graphcanon/graph?tool=defilantech-llmkube