{"data":{"slug":"ethicalml-awesome-production-machine-learning","name":"awesome-production-machine-learning","tagline":"A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning","github_url":"https://github.com/EthicalML/awesome-production-machine-learning","owner":"EthicalML","repo":"awesome-production-machine-learning","owner_avatar_url":"https://avatars.githubusercontent.com/u/43532924?v=4","primary_language":null,"stars":20719,"forks":2585,"topics":["awesome","awesome-list","data-mining","deep-learning","explainability","interpretability","large-scale-machine-learning","large-scale-ml","machine-learning","machine-learning-operations","ml-operations","ml-ops","mlops","privacy-preserving","privacy-preserving-machine-learning","privacy-preserving-ml","production-machine-learning","production-ml","responsible-ai"],"archived":false,"github_pushed_at":"2026-07-03T17:47:07+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/ethicalml-awesome-production-machine-learning","markdown_url":"https://www.graphcanon.com/tools/ethicalml-awesome-production-machine-learning.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/ethicalml-awesome-production-machine-learning","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=ethicalml-awesome-production-machine-learning","description":"A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning","homepage_url":"https://ethicalml.github.io/awesome-production-machine-learning","license":"MIT","open_issues":32,"watchers":416,"ai_summary":null,"readme_excerpt":"## Deployment and Serving\n* [Agenta](https://github.com/Agenta-AI/agenta)  - Agenta provides end-to-end tools for the entire LLMOps workflow: building (LLM playground, evaluation), deploying (prompt and configuration management), and  (LLM observability and tracing).\n* [AirLLM](https://github.com/lyogavin/airllm)  - AirLLM optimizes inference memory usage, allowing 70B large language models to run inference on a single 4GB GPU card without quantization, distillation and pruning.\n* [AITemplate](https://github.com/facebookincubator/AITemplate)  - AITemplate (AIT) is a Python framework that transforms deep neural networks into CUDA (NVIDIA GPU) / HIP (AMD GPU) C++ code for lightning-fast inference serving.\n* [BentoML](https://github.com/bentoml/BentoML)  - BentoML is an open source framework for high performance ML model serving.\n* [BISHENG](https://github.com/dataelement/bisheng)  - BISHENG is an open LLM application devops platform, focusing on enterprise scenarios.\n* [DeepDetect](https://github.com/jolibrain/deepdetect)  - Machine Learning production server for TensorFlow, XGBoost and Cafe models written in C++ and maintained by Jolibrain.\n* [Dynamo](https://github.com/ai-dynamo/dynamo)  - NVIDIA Dynamo is a high-throughput, low-latency inference framework designed for serving generative AI and reasoning models in multi-node distributed environments.\n* [exo](https://github.com/exo-explore/exo)  - exo helps you run your AI cluster at home with everyday devices.\n* [Genkit](https://github.com/firebase/genkit)  - Genkit is an open source framework for building AI-powered apps with familiar code-centric patterns. Genkit makes it easy to develop, integrate, and test AI features with observability and evaluations.\n* [Inference](https://github.com/roboflow/inference)  - A fast, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models. With Inference, you can deploy models such as YOLOv5, YOLOv8, CLIP, SAM, and CogVLM on your own hardware using Docker.\n* [Infinity](https://github.com/michaelfeil/infinity)  - Infinity is a high-throughput, low-latency REST API for serving text-embeddings, reranking models and clip. \n* [IPEX-LLM](https://github.com/intel/ipex-llm)  - IPEX-LLM is a PyTorch library for running LLM on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) with very low latency.\n* [LiteLLM](https://github.com/BerriAI/litellm)  - LiteLLM is a Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq.\n* [LiteRT](https://github.com/google-ai-edge/litert)  - LiteRT (formerly TensorFlow Lite) is Google's high-performance runtime for on-device AI inference, enabling deployment of machine learning models on mobile, embedded, and edge devices.\n* [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM)  - LiteRT-LM is Google's production-ready, high-performance inference framework for deploying Large Language Models on edge devices, with cross-platform support for Android, iOS, Web, Desktop, and IoT.\n* [LitServe](https://github.com/Lightning-AI/LitServe)  - LitServe is a flexible serving engine for AI models built on FastAPI. It supports custom inference engines for models, agents, multi-modal systems, RAG, and complex ML pipelines.\n* [Jina-serve](https://github.com/jina-ai/serve)  - Jina-serve is a framework for building and deploying AI services that communicate via gRPC, HTTP and WebSockets.\n* [Kiln](https://github.com/kiln-ai/kiln)  - Kiln is an OSS tool for fine-tuning LLM models, synthetic data generation, and collaborating on datasets.\n* [KServe](https://github.com/kserve/kserve)  - KServe provides a Kubernetes Custom Resource Definition for serving predictive and generative ML.\n* [KTransformers](https://github.com/kvcache-ai/ktransformers)  - KTransformers is a flexible framework for experiencing cutting-edg","github_created_at":"2018-08-15T14:28:41+00:00","created_at":"2026-07-11T23:39:09.759587+00:00","updated_at":"2026-07-11T23:39:19.927401+00:00","categories":[{"slug":"ai-agents","name":"AI Agents","url":"https://www.graphcanon.com/categories/ai-agents","markdown_url":"https://www.graphcanon.com/categories/ai-agents.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/ai-agents"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"vector-databases","name":"Vector Databases","url":"https://www.graphcanon.com/categories/vector-databases","markdown_url":"https://www.graphcanon.com/categories/vector-databases.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/vector-databases"}],"tags":[{"slug":"awesome","name":"awesome"},{"slug":"awesome-list","name":"awesome-list"},{"slug":"data-mining","name":"data-mining"},{"slug":"deep-learning","name":"deep-learning"},{"slug":"explainability","name":"explainability"},{"slug":"interpretability","name":"interpretability"},{"slug":"large-scale-machine-learning","name":"large-scale-machine-learning"},{"slug":"large-scale-ml","name":"large-scale-ml"}],"trust":{"provenance":{"is_fork":false,"github_id":144863525,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:39:11.551Z","maintenance":{"label":"Active","score":82,"methodology":"github_public_v1","releases_90d":3,"days_since_push":8,"last_release_at":"2026-07-01T04:20:48Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:39:12.045Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:39:11.308Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T23:39:11.308Z"}}}}