---
title: "serve"
type: "tool"
slug: "jina-ai-serve"
canonical_url: "https://www.graphcanon.com/tools/jina-ai-serve"
github_url: "https://github.com/jina-ai/serve"
homepage_url: "https://jina.ai/serve"
stars: 21862
forks: 2242
primary_language: "Python"
license: "Apache-2.0"
categories: ["inference-serving", "llm-frameworks"]
tags: ["llmops", "deep-learning", "cloud-native", "machine-learning", "docker", "generative-ai", "framework", "kubernetes"]
updated_at: "2026-07-07T18:26:08.328888+00:00"
---
# serve
> Build multimodal AI applications with cloud-native stack
Jina-Serve is a framework for building and deploying AI services that communicate via gRPC, HTTP, and WebSockets. It offers native support for major ML frameworks, high-performance service design, LLM serving capabilities, Docker integration, Executor Hub, and one-click deployment to Jina AI Cloud.
## Facts
- Repository: https://github.com/jina-ai/serve
- Homepage: https://jina.ai/serve
- Stars: 21,862 · Forks: 2,242 · Open issues: 25 · Watchers: 210
- Primary language: Python
- License: Apache-2.0
- Last pushed: 2025-03-24T13:59:54+00:00
## Categories
- [Inference & Serving](/categories/inference-serving.md)
- [LLM Frameworks](/categories/llm-frameworks.md)
## Tags
llmops, deep-learning, cloud-native, machine-learning, docker, generative-ai, framework, kubernetes
## Related tools
- [ollama](/tools/ollama-ollama.md) - Get up and running with Kimi-K2.6, GLM-5.1, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models. (★ 175,659)
- [prompts.chat](/tools/f-prompts-chat.md) - The world's largest open-source prompt library for AI (★ 165,019)
- [transformers](/tools/huggingface-transformers.md) - 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models (★ 162,347)
- [open-webui](/tools/open-webui-open-webui.md) - User-friendly AI Interface (Supports Ollama, OpenAI API, ...) (★ 144,575)
- [awesome-llm-apps](/tools/shubhamsaboo-awesome-llm-apps.md) - 100+ AI Agent & RAG apps you can actually run — clone, customize, ship. (★ 116,702)
- [LLMs-from-scratch](/tools/rasbt-llms-from-scratch.md) - Implement a ChatGPT-like LLM in PyTorch from scratch (★ 98,711)
- [TradingAgents](/tools/tauricresearch-tradingagents.md) - TradingAgents: Multi-Agents LLM Financial Trading Framework (★ 91,610)
- [caveman](/tools/juliusbrussee-caveman.md) - Cuts 65% of tokens in AI coding agent responses. (★ 86,150)
## README (excerpt)
```text
# Jina-Serve
Jina-serve is a framework for building and deploying AI services that communicate via gRPC, HTTP and WebSockets. Scale your services from local development to production while focusing on your core logic.
## Key Features
- Native support for all major ML frameworks and data types
- High-performance service design with scaling, streaming, and dynamic batching
- LLM serving with streaming output
- Built-in Docker integration and Executor Hub
- One-click deployment to Jina AI Cloud
- Enterprise-ready with Kubernetes and Docker Compose support
Comparison with FastAPI
Key advantages over FastAPI:
- DocArray-based data handling with native gRPC support
- Built-in containerization and service orchestration
- Seamless scaling of microservices
- One-command cloud deployment
## Install
```bash
pip install jina
```
See guides for [Apple Silicon](https://jina.ai/serve/get-started/install/apple-silicon-m1-m2/) and [Windows](https://jina.ai/serve/get-started/install/windows/).
## Core Concepts
Three main layers:
- **Data**: BaseDoc and DocList for input/output
- **Serving**: Executors process Documents, Gateway connects services
- **Orchestration**: Deployments serve Executors, Flows create pipelines
## Build AI Services
Let's create a gRPC-based AI service using StableLM:
```python
from jina import Executor, requests
from docarray import DocList, BaseDoc
from transformers import pipeline
class Prompt(BaseDoc):
text: str
class Generation(BaseDoc):
prompt: str
text: str
class StableLM(Executor):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.generator = pipeline(
'text-generation', model='stabilityai/stablelm-base-alpha-3b'
)
@requests
def generate(self, docs: DocList[Prompt], **kwargs) -> DocList[Generation]:
generations = DocList[Generation]()
prompts = docs.text
llm_outputs = self.generator(prompts)
for prompt, output in zip(prompts, llm_outputs):
generations.append(Generation(prompt=prompt, text=output))
return generations
```
Deploy with Python or YAML:
```python
from jina import Deployment
from executor import StableLM
dep = Deployment(uses=StableLM, timeout_ready=-1, port=12345)
with dep:
dep.block()
```
```yaml
jtype: Deployment
with:
uses: StableLM
py_modules:
- executor.py
timeout_ready: -1
port: 12345
```
Use the client:
```python
from jina import Client
from docarray import DocList
from executor import Prompt, Generation
prompt = Prompt(text='suggest an interesting image generation prompt')
client = Client(port=12345)
response = client.post('/', inputs=[prompt], return_type=DocList[Generation])
```
## Build Pipelines
Chain services into a Flow:
```python
from jina import Flow
flow = Flow(port=12345).add(uses=StableLM).add(uses=TextToImage)
with flow:
flow.block()
```
## Scaling and Deployment
### Local Scaling
Boost throughput with built-in features:
- Replicas for parallel processing
- Shards for data partitioning
- Dynamic batching for efficient model inference
Example scaling a Stable Diffusion deployment:
```yaml
jtype: Deployment
with:
uses: TextToImage
timeout_ready: -1
py_modules:
- text_to_image.py
env:
CUDA_VISIBLE_DEVICES: RR
replicas: 2
uses_dynamic_batc
```
---
**Machine-readable endpoints**
- JSON: [`/api/graphcanon/tools/jina-ai-serve`](/api/graphcanon/tools/jina-ai-serve)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)
_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_