“inference”

Large Language Model Text Generation Inference

The easiest way to serve AI apps and models

OptiLLM is an OpenAI API-compatible optimizing inference proxy that enhances LLM accuracy and performance on reasoning tasks without needing to train or fine-tune models.

A high-throughput and memory-efficient inference and serving engine for LLMs

High-speed Large Language Model Serving for Local Deployment

LLM inference in C/C++

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Self-Hosting LLMs Made Easy

Highly Performant, Modular and Memory Safe Ingestion, Inference and Indexing in Rust

End-to-end framework for building custom AI applications and agents.

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

Artificial Intelligence Controller Interface (AICI)

Repository listing free and trial-based LLM inference API resources

Machine Learning Engineering Open Book

Community maintained hardware plugin for vLLM on Ascend

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models

Unified framework for scaling AI and Python applications

A generative speech model for daily dialogue

SGLang high-performance serving framework

Find secrets with Gitleaks

分享大模型技术原理与实战经验，涵盖工程化和应用落地

Repository for running large language models with reduced memory usage on limited GPU hardware.

20+ high-performance LLMs with recipes to pretrain, finetune, and deploy at scale.

AI-native proxy server and data plane for agentic apps

Cube Studio开源一站式云原生机器学习平台

Implement a reasoning LLM in PyTorch from scratch, step by step

CubeStudio是开源云原生一站式机器学习/深度学习/AI平台

Neuron is an Agentic Framework in PHP for building production-ready AI driven applications.

Simple, Explicit and Transparent LLM Applications

VectorHub: A Learning Hub for Vector Retrieval in ML

User-friendly AI Interface (Supports Ollama, OpenAI API, ...)

Universal LLM Deployment Engine with ML Compilation

Reference stack for running AI agents securely in OpenShell

An Open Bilingual Chat LLM

Run LLMs at home, BitTorrent-style

Summary of the world's best LLM resources.

Postgres with GPUs for ML/AI applications.

Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice