Home/Inference & Serving/sarathi-serve

sarathi-serve

microsoft/sarathi-serve

A low-latency and high-throughput serving engine for LLMs

GitHub

GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026

36views this month

527 stars67 forksLast push Jan 8, 2026 Python Apache-2.0

Decision brief

Sarathi Serve targets efficient low-latency and high-throughput inference for LLMs using Python.

Good fit when

Optimize Python-based projects needing quick responses from large language models.
Demand high throughput alongside minimal inference delay.

Avoid when

Necessitate a non-Python environment for deployment and operation.
Prefer a tool that incorporates more than just low-latency, high-throughput focus such as multi-language support or specialized optimizations.

Observed Jul 17, 2026 · Source: enrich:decision_facts

sarathi-serve alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Slowing (254d since push); As of Sep 20, 2026
Provenance: Not a fork · Organization account; As of Sep 20, 2026
Security (OSV): No lockfile; As of Jul 11, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Backing

Company context for Microsoft. Display-only - separate from trust and ranking.

Company: Microsoft·GitHub org profile·Jul 11, 2026
Employees: 221,000·Wikidata (P1128 employees)·Jul 11, 2026
Commercial model: Pure OSS·GitHub org profile (public repos)·Jul 11, 2026

Install

pip install sarathi-serve

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

omlx

LLM inference server with continuous batching and SSD caching for Apple Silicon

22k★

whichllm

Command-line tool to find and benchmark local LLM performance

6.7k★

flashinfer

FlashInfer is a kernel library for serving large language models

6.5k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

Sarathi Serve is a Python-based project designed to provide efficient inference services with emphasis on low latency and high throughput for large language models.

Capability facts

Languages: python
Source: github.language+pyproject.toml · Sep 20, 2026

Graph entities

Microsoftcompany

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Sep 20, 2026)

pip install -e .

Source link

README

Install Sarathi Serve

For agents

This page has a .md twin and JSON over the API.

/tools/microsoft-sarathi-serve.md GET /api/graphcanon/tools/microsoft-sarathi-serve GET /api/graphcanon/graph?tool=microsoft-sarathi-serve