{"data":{"slug":"kubeai-project-kubeai","name":"kubeai","tagline":"AI Inference Operator for Kubernetes","github_url":"https://github.com/kubeai-project/kubeai","owner":"kubeai-project","repo":"kubeai","owner_avatar_url":"https://avatars.githubusercontent.com/u/232319222?v=4","primary_language":"Go","stars":1265,"forks":137,"topics":["ai","autoscaler","faster-whisper","inference-operator","k8s","kubernetes","llm","ollama","ollama-operator","openai-api","vllm","vllm-operator","whisper"],"archived":false,"github_pushed_at":"2026-09-17T23:56:08+00:00","maintenance_label":"Very active","stars_delta_30d":28,"url":"https://www.graphcanon.com/tools/kubeai-project-kubeai","markdown_url":"https://www.graphcanon.com/tools/kubeai-project-kubeai.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/kubeai-project-kubeai","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=kubeai-project-kubeai","description":"AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.","homepage_url":"https://www.kubeai.org","license":"Apache-2.0","open_issues":121,"watchers":13,"ai_summary":"KubeAI is an AI inference operator for Kubernetes that simplifies the deployment and scaling of machine learning models, including LLMs, embeddings, and speech-to-text services.","readme_excerpt":"# KubeAI: AI Inferencing Operator\n\n<p align=\"left\">\n  <img src=\"https://img.shields.io/github/license/kubeai-project/kubeai\"/>\n  <img src=\"https://img.shields.io/github/go-mod/go-version/kubeai-project/kubeai\"/>\n  <img src=\"https://img.shields.io/github/stars/kubeai-project/kubeai\"/>\n  <img src=\"https://img.shields.io/github/contributors/kubeai-project/kubeai\" />\n  <img src=\"https://img.shields.io/github/last-commit/kubeai-project/kubeai/main\" />\n</p>\n\nDeploy and scale machine learning models on Kubernetes. \n\nBuilt for LLMs, embeddings, reranking and speech-to-text.\n\n## Highlights\n\nWhat is it for?\n\n🚀 **LLM Inferencing** - Operate vLLM and Ollama servers  \n🎙️ **Speech Processing** - Transcribe audio with FasterWhisper  \n🔢 **Vector Embeddings** - Generate embeddings with Infinity  \n📚 **Reranking** - Reorder search results with cross-encoder models  \n\nWhat do you get?\n\n⚡️ **Intelligent Scaling** - Scale from zero to meet demand  \n📊 **Optimized Routing** - Dramatically improves performance at scale ([see paper](./blog/posts/llm-load-balancing-at-scale-chwbl.md))  \n💾 **Model Caching** - Automates downloading & mounting (EFS, etc.)  \n🧩 **Dynamic Adapters** - Orchestrates LoRA adapters across replicas  \n📨 **Event Streaming** - Integrates with Kafka, PubSub, and more  \n\nWe strive for an \"it justs works\" experience:\n\n🔗 **OpenAI Compatible** - Works with OpenAI client libraries  \n🛠️ **Zero Dependencies** - Does not require Istio, Knative, etc.  \n🖥 **Hardware Flexible** - Runs on CPU, GPU, or TPU  \n\nQuotes from the community:\n\n> reusable, well abstracted solution to run LLMs - [Mike Ensor](https://www.linkedin.com/posts/mikeensor_gcp-solutions-public-retail-edge-available-cluster-traits-activity-7237515920259104769-vBs9?utm_source=share&utm_medium=member_desktop), Google\n\n## Why KubeAI?\n\n### Better performance at scale\n\nWhen running multiple replicas of vLLM, the random load balancing strategy built into kube-proxy that backs standard Kubernetes Services performs poorly (TTFT & throughput). This is because vLLM isn't stateless, its performance is heavily influenced by the state of its KV cache.\n\nThe KubeAI proxy includes a prefix-aware load balancing strategy that optimizes KV cache utilization - resulting in dramatic improvements to overall system performance.\n\n<img src=\"./graphs/ttft-benchmark.png\" width=\"80%\"/>\n\nSee the [full paper](./blog/posts/llm-load-balancing-at-scale-chwbl.md) for more details.\n\n### Simplicity and ease of use\n\nKubeAI does not depend on other systems like Istio & Knative (for scale-from-zero), or the Prometheus metrics adapter (for autoscaling). This allows KubeAI to work out of the box in almost any Kubernetes cluster. Day-two operations is greatly simplified as well - don't worry about inter-project version and configuration mismatches.\n\nThe project ships with a catalog of popular models, pre-configured for common GPU types. This means you can spend less time tweaking vLLM-specific flags. As we expand, we plan to build out an extensive model optimization pipeline that will ensure you get the most out of your hardware.\n\n### OpenAI API Compatibility\n\nNo need to change your client libraries, KubeAI supports the following endpoints:\n\n```bash\n/v1/chat/completions\n/v1/completions\n/v1/embeddings\n/v1/rerank\n/v1/models\n/v1/audio/transcriptions\n```\n\n## Architecture\n\nKubeAI consists of two primary sub-components:\n\n**1. The model proxy:** the KubeAI proxy provides an OpenAI-compatible API. Behind this API, the proxy implements a prefix-aware load balancing strategy that optimizes for KV the cache utilization of the backend serving engines (i.e. vLLM). The proxy also implements request queueing (while the system scales from zero replicas) and request retries (to seamlessly handle bad backends).\n\n**2. The model operator:** the KubeAI model operator manages backend server Pods directly. It automates common operations such as downloading models, mounting volumes, and loading dynamic LoRA adapters via the KubeAI Mode","github_created_at":"2023-10-21T00:59:51+00:00","created_at":"2026-07-11T23:13:16.104029+00:00","updated_at":"2026-09-20T04:30:05.380091+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"},{"slug":"speech-audio","name":"Speech & Audio","url":"https://www.graphcanon.com/categories/speech-audio","markdown_url":"https://www.graphcanon.com/categories/speech-audio.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/speech-audio"}],"tags":[{"slug":"ai","name":"ai"},{"slug":"autoscaler","name":"autoscaler"},{"slug":"faster-whisper","name":"faster-whisper"},{"slug":"inference-operator","name":"inference-operator"},{"slug":"k8s","name":"k8s"},{"slug":"kubernetes","name":"kubernetes"},{"slug":"llm","name":"llm"},{"slug":"ollama","name":"ollama"}],"trust":{"provenance":{"is_fork":false,"github_id":707913607,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:21:02.745Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":5,"days_since_push":0,"last_release_at":"2026-07-30T09:53:09Z","stars_delta_30d":28,"open_issues_delta_30d":9},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":41,"high_count":0,"last_scan_at":"2026-09-18T08:21:04.877Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:21:01.662Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-09-18T08:21:01.662Z","managed_saas":false},"languages":{"value":["go"],"source":"github.language","observed_at":"2026-09-18T08:21:01.662Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-09-18T08:21:01.662Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:21:01.662Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium","summary":"The tool is free and open-source under the Apache-2.0 license, but additional services or support might incur costs"},"requirements":{"notes":["Requires a Kubernetes cluster","Supports deployment on CPU, GPU, or TPU hardware"],"min_ram_gb":4,"requires_docker":true},"constraints":{"min_ram_gb":4,"pricing_model":"freemium","requires_docker":true},"when_to_use":["you need to deploy and scale machine learning models on Kubernetes, especially LLMs, embeddings, and speech-to-text services","you require intelligent scaling and optimized routing for large-scale deployments","you want an OpenAI-compatible API without changing your client libraries","you prefer a solution that does not depend on additional systems like Istio or Knative"],"when_not_to_use":["you are looking for a solution that requires additional systems like Istio or Knative for scale-from-zero","you need a tool that heavily relies on Prometheus metrics adapter for autoscaling","you are working in an environment that does not support Kubernetes","you prefer a more complex setup with multiple dependencies for customization"],"source":"enrich:decision_facts","observed_at":"2026-09-18T10:52:57.867Z"},"constraint_facets":{"min_ram_gb":4,"pricing_model":"freemium","requires_docker":true},"decision_summary":[{"label":"Pricing","value":"freemium - The tool is free and open-source under the Apache-2.0 license, but additional services or support might incur costs"},{"label":"Requirements","value":"Min 4 GB RAM; Requires Docker; Requires a Kubernetes cluster; Supports deployment on CPU, GPU, or TPU hardware"},{"label":"Adopt for","value":"KubeAI is an AI inference operator for Kubernetes that simplifies the deployment and scaling of machine learning models, including LLMs, embeddings, and speech-to-text services, with intelligent scaling and optimized for"},{"label":"License detail","value":"Apache-2.0"}]}}