“serving”

A high-throughput and memory-efficient inference and serving engine for LLMs

The easiest way to serve AI apps and models

Build multimodal AI applications with cloud-native stack

Unified framework for scaling AI and Python applications

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

SGLang high-performance serving framework

Community maintained hardware plugin for vLLM on Ascend

Artificial Intelligence Controller Interface (AICI)

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

High-speed Large Language Model Serving for Local Deployment

The AI search platform

Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice

LLM/VLM load balancer and serving platform built around the ggml ecosystem

API server for serving LangChain models and pipelines with FastApi

分享大模型技术原理与实战经验，涵盖工程化和应用落地

Self-Hosting LLMs Made Easy

ClearML - Auto-Magical CI/CD for AI workloads

End-to-end framework for building custom AI applications and agents.

Reproducible development environment for humans and agents

Multi-modal vector database supporting unified SQL access to structured and unstructured data

Repository for prompt engineering resources, guides, and tools focused on language models (LLMs).

Universal memory layer for AI Agents

Stealth Chromium that passes every bot detection test.

Repository for preparing AI/ML technical interviews with chapters on general coding, ML coding, fundamentals/breadth (including LLMs and multimodal AI), system design, behavioral questions, and more.

Community developed SQL server with advanced features and performance.

Curated list of LLMOps tools for developers

Repository collecting GUI clients for ChatGPT, Claude, and other LLMs

Attu - The AI Workbench for Milvus

The Virtual Feature Store. Turn your existing data infrastructure into a feature store.

AI-native HTAP database with Git-for-Data and built-in vector search

Unified multimodal backend for AI data apps