Home/Inference & Serving/JetStream

JetStream

AI-Hypercomputer/JetStream

Throughput and memory optimized engine for LLM inference on XLA devices

GitHub

GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026

27views this month

458 stars67 forksLast push Jan 5, 2026 Python Apache-2.0

Decision brief

JetStream optimises throughput and memory for LLM inference on XLA devices like TPUs, with potential GPU support in future.

Good fit when

* You are working with large language models (LLMs) that require efficient inference on hardware supported by XLA, particularly TPUs.
* Your project benefits from optimised memory usage and high throughput performance for inference tasks.

Avoid when

* If your primary compute platform is not an XLA-compatible device such as TPU; JetStream's current focus is on systems that are supported by XLA.
* When you need immediate support for GPUs, since GPU functionality is marked as a future potential enhancement.

Observed Jul 17, 2026 · Source: enrich:decision_facts

JetStream alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Slowing (257d since push); As of Sep 20, 2026
Provenance: Not a fork · Organization account; As of Sep 20, 2026
Security (OSV): No lockfile; As of Jul 11, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install JetStream

PyPI

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

92k★

TensorRT-LLM

Python API for defining and optimizing Large Language Models for efficient inference on NVIDIA GPUs

15k★

mistral.rs

Fast flexible LLM inference

7.7k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

JetStream is designed to optimize throughput and memory usage for the inference of large language models (LLMs) running on XLA-accelerated hardware like TPUs, with potential support for GPUs in future releases.

Capability facts

Languages: python
Source: github.language · Sep 20, 2026

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Python runtimePython

Source: README excerpt (regex_v1, Sep 20, 2026)

python -m jetstream.core.implementations.mock.server

Source link

README

[!WARNING] Notice of Archival: In an effort to streamline TPU inference efforts in open source, we have migrated core functionality in Jetstream to the new tpu inference repository. For this reason, we will be archiving Jetstream on February 1st 2026. Please note, archival does n...

For agents

This page has a .md twin and JSON over the API.

/tools/ai-hypercomputer-jetstream.md GET /api/graphcanon/tools/ai-hypercomputer-jetstream GET /api/graphcanon/graph?tool=ai-hypercomputer-jetstream