Inference & Serving

transformers

Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models

open-webui

User-friendly AI Interface (Supports Ollama, OpenAI API, ...)

ollama

Get up and running with various large language models using Ollama.

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

claude-mem

Persistent Context Across Sessions for Every Agent

unsloth

A web UI for training and running open models locally.

anything-llm

Self-hosted agent experience with deployment scripts for multiple environments

JeecgBoot

AI低代码平台，实现快速生成前后端系统及模块

litellm

Python SDK and Proxy Server for calling multiple LLM APIs

OmniRoute

Free AI gateway with multi-provider support and token savings

ray

Ray is an AI compute engine with a core distributed runtime and AI Libraries for accelerating ML workloads.

LibreChat

Enhanced ChatGPT Clone with extensive features and integrations for self-hosting

sglang

High-performance serving framework for large language and multimodal models

whisper.cpp

Port of OpenAI's Whisper model in C/C++ for speech-to-text inference

mlflow

AI engineering platform for debugging, evaluating, monitoring, and optimizing AI applications

netron

Visualizer for neural network, deep learning and machine learning models

ruflo

The leading agent meta-harness for intelligent multi-player swarms and autonomous workflows

9router

Unlimited FREE AI coding with auto-fallback and token savings

ncnn

High-performance neural network inference framework optimized for mobile platforms

screenpipe

AI that records and analyzes everything you do, say, hear locally

ai-guide

免费开放的AI知识共享平台

jan

open source alternative to ChatGPT that runs offline locally

ml-engineering

Machine Learning Engineering Open Book

airllm

AirLLM 70B inference with single 4GB GPU

generative-ai

Sample code and notebooks for Generative AI on Google Cloud, with Gemini Enterprise Agent Platform

openvino

OpenVINO is an open source toolkit for optimizing and deploying AI inference.

DeepSpeed

Deep learning optimization library for efficient distributed training and inference

omlx

LLM inference server with continuous batching and SSD caching for Apple Silicon

pytorch

Tensors and Dynamic neural networks in Python with strong GPU acceleration

OpenLLM

Run any open-source LLMs as OpenAI compatible API endpoint in the cloud.

inference

Unified production-ready inference API for various models

awesome-LLM-resources

Summary of the world's best LLM resources.

oumi

Easily fine-tune, evaluate and deploy open source LLMs/VLMs

dynamo

A Datacenter Scale Distributed Inference Serving Framework

runanywhere-sdks

Production ready toolkit to run AI locally

CosyVoice

Multi-lingual large voice generation model with full-stack abilities for inference, training and deployment.

train-llm-from-scratch

A straightforward method for training your LLM from raw text to aligned model generation

awesome-free-llm-apis

List of Permanent Free LLM API

plano

An AI-native proxy and data plane for agentic apps

BentoML

The easiest way to serve AI apps and models

serving

A flexible, high-performance serving system for machine learning models

llama.cpp

LLM inference in C/C++

MNN

Blazing-fast, lightweight inference engine for high-performance on-device LLMs and Edge AI

lmdeploy

Toolkit for compressing, deploying, and serving LLMs

metaflow

Build, Manage and Deploy AI/ML Systems

llamafile

Distribute and run LLMs with a single file.

llm-universe

面向小白开发者的LLM应用开发教程

shell_gpt

A command-line productivity tool powered by AI large language models

llm-action

Aims to share large model technology principles and practical experience (large model engineering, application implementation)

bifrost

Fast Enterprise AI Gateway with Adaptive Load Balancer and Guardrails

TensorRT-LLM

Python API for defining and optimizing Large Language Models (LLMs) on NVIDIA GPUs

mcp

Open source MCP Servers for AWS

litgpt

High-performance LLMs with recipes for pretraining, finetuning and deployment

self-llm

A guide for fine-tuning and deploying open-source large language models tailored for a Chinese audience on Linux.

llm

Access large language models from the command-line

pytorch-lightning

Pretrain, finetune ANY AI model of ANY size on 1 or 10,000+ GPUs with zero code changes.

bitsandbytes

Large language model quantization toolkit for PyTorch.

go-stock

AI-empowered stock analysis tool for multiple markets

private-gpt

Complete API layer for private AI applications on local models

llm-engineer-toolkit

A curated list of over 120 LLM libraries categorized.