awesome-production-machine-learning

Deployment and Serving

• Agenta - Agenta provides end-to-end tools for the entire LLMOps workflow: building (LLM playground, evaluation), deploying (prompt and configuration management), and (LLM observability and tracing).
• AirLLM - AirLLM optimizes inference memory usage, allowing 70B large language models to run inference on a single 4GB GPU card without quantization, distillation and pruning.
• AITemplate - AITemplate (AIT) is a Python framework that transforms deep neural networks into CUDA (NVIDIA GPU) / HIP (AMD GPU) C++ code for lightning-fast inference serving.
• BentoML - BentoML is an open source framework for high performance ML model serving.
• BISHENG - BISHENG is an open LLM application devops platform, focusing on enterprise scenarios.
• DeepDetect - Machine Learning production server for TensorFlow, XGBoost and Cafe models written in C++ and maintained by Jolibrain.
• Dynamo - NVIDIA Dynamo is a high-throughput, low-latency inference framework designed for serving generative AI and reasoning models in multi-node distributed environments.
• exo - exo helps you run your AI cluster at home with everyday devices.
• Genkit - Genkit is an open source framework for building AI-powered apps with familiar code-centric patterns. Genkit makes it easy to develop, integrate, and test AI features with observability and evaluations.
• GoModel - GoModel is a self-hosted AI gateway written in Go that exposes a unified OpenAI-compatible API across OpenAI, Anthropic, Gemini, Groq, xAI, Ollama and other providers, with routing, usage tracking, rate limits, and guardrails.
• Inference - A fast, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models. With Inference, you can deploy models such as YOLOv5, YOLOv8, CLIP, SAM, and CogVLM on your own hardware using Docker.
• Infinity - Infinity is a high-throughput, low-latency REST API for serving text-embeddings, reranking models and clip.
• IPEX-LLM - IPEX-LLM is a PyTorch library for running LLM on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) with very low latency.
• LiteLLM - LiteLLM is a Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq.
• LiteRT - LiteRT (formerly TensorFlow Lite) is Google's high-performance runtime for on-device AI inference, enabling deployment of machine learning models on mobile, embedded, and edge devices.
• LiteRT-LM - LiteRT-LM is Google's production-ready, high-performance inference framework for deploying Large Language Models on edge devices, with cross-platform support for Android, iOS, Web, Desktop, and IoT.
• LitServe - LitServe is a flexible serving engine for AI models built on FastAPI. It supports custom inference engines for models, agents, multi-modal systems, RAG, and complex ML pipelines.
• Jina-serve - Jina-serve is a framework for building and deploying AI services that communicate via gRPC, HTTP and WebSockets.
• Kiln - Kiln is an OSS tool for fine-tuning LLM models, synthetic data generation, and collaborating on dataset