Home/Inference & Serving/sarathi-serve

sarathi-serve

microsoft/sarathi-serve

A low-latency and high-throughput serving engine for LLMs

GitHub

GraphCanon updated today · GitHub synced today · 26 views this month

520 stars65 forksLast push 7mo Python Apache-2.0

Decision brief

Sarathi Serve targets efficient low-latency and high-throughput inference for LLMs using Python.

Good fit when

Optimize Python-based projects needing quick responses from large language models.
Demand high throughput alongside minimal inference delay.

Avoid when

Necessitate a non-Python environment for deployment and operation.
Prefer a tool that incorporates more than just low-latency, high-throughput focus such as multi-language support or specialized optimizations.

Observed Jul 17, 2026 · Source: enrich:decision_facts

sarathi-serve alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Slowing (229d since push); As of today
Provenance: Not a fork · Organization account; As of today
Security (OSV): No lockfile; As of 1mo

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Backing

Company context for Microsoft. Display-only - separate from trust and ranking.

Company: Microsoft·GitHub org profile·1mo
Employees: 221,000·Wikidata (P1128 employees)·1mo
Commercial model: Pure OSS·GitHub org profile (public repos)·1mo

Install

pip install sarathi-serve

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

omlx

LLM inference server with continuous batching and SSD caching for Apple Silicon

19k★

serving

A flexible, high-performance serving system for machine learning models

6.4k★

flashinfer

FlashInfer is a kernel library for serving large language models

6.2k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

Sarathi Serve is a Python-based project designed to provide efficient inference services with emphasis on low latency and high throughput for large language models.

Capability facts

Languages: python
Source: github.language+pyproject.toml · Aug 25, 2026

Graph entities

Microsoftcompany

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Aug 25, 2026)

pip install -e .

Source link

README

Install Sarathi-Serve

pip install -e .

For agents

This page has a .md twin and JSON over the API.

/tools/microsoft-sarathi-serve.md GET /api/graphcanon/tools/microsoft-sarathi-serve GET /api/graphcanon/graph?tool=microsoft-sarathi-serve