{"data":{"slug":"kubeai-project-kubeai","name":"kubeai","tagline":"AI Inference Operator for Kubernetes","github_url":"https://github.com/kubeai-project/kubeai","owner":"kubeai-project","repo":"kubeai","owner_avatar_url":"https://avatars.githubusercontent.com/u/232319222?v=4","primary_language":"Go","stars":1222,"forks":128,"topics":["ai","autoscaler","faster-whisper","inference-operator","k8s","kubernetes","llm","ollama","ollama-operator","openai-api","vllm","vllm-operator","whisper"],"archived":false,"github_pushed_at":"2026-07-10T10:21:44+00:00","maintenance_label":"Very active","url":"https://www.graphcanon.com/tools/kubeai-project-kubeai","markdown_url":"https://www.graphcanon.com/tools/kubeai-project-kubeai.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/kubeai-project-kubeai","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=kubeai-project-kubeai","description":"AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.","homepage_url":"https://www.kubeai.org","license":"Apache-2.0","open_issues":120,"watchers":13,"ai_summary":"Serves machine learning models in production on Kubernetes, supporting LLMs, embeddings, speech-to-text, and more.","readme_excerpt":"# KubeAI: AI Inferencing Operator\n\n<p align=\"left\">\n  <img src=\"https://img.shields.io/github/license/kubeai-project/kubeai\"/>\n  <img src=\"https://img.shields.io/github/go-mod/go-version/kubeai-project/kubeai\"/>\n  <img src=\"https://img.shields.io/github/stars/kubeai-project/kubeai\"/>\n  <img src=\"https://img.shields.io/github/contributors/kubeai-project/kubeai\" />\n  <img src=\"https://img.shields.io/github/last-commit/kubeai-project/kubeai/main\" />\n</p>\n\nDeploy and scale machine learning models on Kubernetes. \n\nBuilt for LLMs, embeddings, reranking and speech-to-text.\n\n## Highlights\n\nWhat is it for?\n\n🚀 **LLM Inferencing** - Operate vLLM and Ollama servers  \n🎙️ **Speech Processing** - Transcribe audio with FasterWhisper  \n🔢 **Vector Embeddings** - Generate embeddings with Infinity  \n📚 **Reranking** - Reorder search results with cross-encoder models  \n\nWhat do you get?\n\n⚡️ **Intelligent Scaling** - Scale from zero to meet demand  \n📊 **Optimized Routing** - Dramatically improves performance at scale ([see paper](./blog/posts/llm-load-balancing-at-scale-chwbl.md))  \n💾 **Model Caching** - Automates downloading & mounting (EFS, etc.)  \n🧩 **Dynamic Adapters** - Orchestrates LoRA adapters across replicas  \n📨 **Event Streaming** - Integrates with Kafka, PubSub, and more  \n\nWe strive for an \"it justs works\" experience:\n\n🔗 **OpenAI Compatible** - Works with OpenAI client libraries  \n🛠️ **Zero Dependencies** - Does not require Istio, Knative, etc.  \n🖥 **Hardware Flexible** - Runs on CPU, GPU, or TPU  \n\nQuotes from the community:\n\n> reusable, well abstracted solution to run LLMs - [Mike Ensor](https://www.linkedin.com/posts/mikeensor_gcp-solutions-public-retail-edge-available-cluster-traits-activity-7237515920259104769-vBs9?utm_source=share&utm_medium=member_desktop), Google\n\n## Why KubeAI?\n\n### Better performance at scale\n\nWhen running multiple replicas of vLLM, the random load balancing strategy built into kube-proxy that backs standard Kubernetes Services performs poorly (TTFT & throughput). This is because vLLM isn't stateless, its performance is heavily influenced by the state of its KV cache.\n\nThe KubeAI proxy includes a prefix-aware load balancing strategy that optimizes KV cache utilization - resulting in dramatic improvements to overall system performance.\n\n<img src=\"./graphs/ttft-benchmark.png\" width=\"80%\"/>\n\nSee the [full paper](./blog/posts/llm-load-balancing-at-scale-chwbl.md) for more details.\n\n### Simplicity and ease of use\n\nKubeAI does not depend on other systems like Istio & Knative (for scale-from-zero), or the Prometheus metrics adapter (for autoscaling). This allows KubeAI to work out of the box in almost any Kubernetes cluster. Day-two operations is greatly simplified as well - don't worry about inter-project version and configuration mismatches.\n\nThe project ships with a catalog of popular models, pre-configured for common GPU types. This means you can spend less time tweaking vLLM-specific flags. As we expand, we plan to build out an extensive model optimization pipeline that will ensure you get the most out of your hardware.\n\n### OpenAI API Compatibility\n\nNo need to change your client libraries, KubeAI supports the following endpoints:\n\n```bash\n/v1/chat/completions\n/v1/completions\n/v1/embeddings\n/v1/rerank\n/v1/models\n/v1/audio/transcriptions\n```\n\n## Architecture\n\nKubeAI consists of two primary sub-components:\n\n**1. The model proxy:** the KubeAI proxy provides an OpenAI-compatible API. Behind this API, the proxy implements a prefix-aware load balancing strategy that optimizes for KV the cache utilization of the backend serving engines (i.e. vLLM). The proxy also implements request queueing (while the system scales from zero replicas) and request retries (to seamlessly handle bad backends).\n\n**2. The model operator:** the KubeAI model operator manages backend server Pods directly. It automates common operations such as downloading models, mounting volumes, and loading dynamic LoRA adapters via the KubeAI Mode","github_created_at":"2023-10-21T00:59:51+00:00","created_at":"2026-07-11T23:13:16.104029+00:00","updated_at":"2026-07-12T04:10:32.159219+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"speech-audio","name":"Speech & Audio","url":"https://www.graphcanon.com/categories/speech-audio","markdown_url":"https://www.graphcanon.com/categories/speech-audio.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/speech-audio"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"llm","name":"llm"},{"slug":"ai","name":"ai"},{"slug":"autoscaler","name":"autoscaler"},{"slug":"faster-whisper","name":"faster-whisper"},{"slug":"inference-operator","name":"inference-operator"},{"slug":"k8s","name":"k8s"},{"slug":"kubernetes","name":"kubernetes"},{"slug":"ollama","name":"ollama"}],"trust":{"provenance":{"is_fork":false,"github_id":707913607,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:13:25.091Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1,"last_release_at":"2026-03-31T11:54:18Z"},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":36,"high_count":0,"last_scan_at":"2026-07-11T23:13:25.572Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-12T04:10:08.484Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-07-12T04:10:08.484Z","managed_saas":false},"languages":{"value":["go"],"source":"github.language","observed_at":"2026-07-12T04:10:08.484Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-07-12T04:10:08.484Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-07-12T04:10:08.484Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["- When you need to operate vLLM and Ollama servers for LLM inferencing","- For speech processing with audio transcription using FasterWhisper","- If you require generating vector embeddings without managing dependencies like Istio or Knative","- To leverage optimized scaling and intelligent routing, especially when deploying large language models which benefit from prefix-aware load balancing"],"when_not_to_use":["- When your setup requires non-standard Kubernetes services that mandate the use of Istio or similar dependency injection systems","- If you're working in a constrained environment where zero-dependency is not desirable due to specific requirements for extended observability tools like Prometheus"],"source":"enrich:decision_facts","observed_at":"2026-07-12T04:10:31.978Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"kubeai is an AI Inference Operator for Kubernetes that simplifies serving ML models in production environments and optimizes performance at scale."}]}}