GraphCanon updated 2w · GitHub synced 2w
Decision brief
Serve offers dedicated support for deploying and scaling PyTorch models with features tailored towards large language model deployment, such as integration with Hugging Face.
Good fit when
- If you are working primarily with PyTorch-based machine-learning projects that require production-grade deployments.
- For teams needing to optimize the performance of their models through integration with hardware like GPUs and frameworks like Kubernetes.
Avoid when
- Avoid if your primary model development is not in PyTorch or requires deployment using a language other than Java.
- Not suitable if you do not require the fine-grained control and optimization provided by tools such as VLLM or TensorRT-LLM.
Observed Jul 16, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Archived (360d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
git clone https://github.com/pytorch/serveSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
A toolkit for deploying and scaling PyTorch machine-learning models in production environments with support for LLM deployment and optimization.
Capability facts
- Languages
- java
Source: github.language · Aug 2, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Source: README excerpt (regex_v1, Aug 2, 2026)
python ./ts_scripts/install_dependencies.pySource link
Tags
README
🚀 Quick start with TorchServe
---
# Install dependencies
python ./ts_scripts/install_dependencies.py
---
### 🚀 Quick start with TorchServe (conda)
```bash
---
### 🤖 Quick Start LLM Deployment
#### VLLM Engine
```bash
---
# Make sure to install torchserve with pip or conda as described above and login with `huggingface-cli login`
python -m ts.llm_launcher --model_id meta-llama/Llama-3.2-3B-Instruct --disable_token_auth
---
# Make sure to install torchserve with python venv as described above and login with `huggingface-cli login`
---
# pip install -U --use-deprecated=legacy-resolver -r requirements/trt_llm.txt
python -m ts.llm_launcher --model_id meta-llama/Meta-Llama-3.1-8B-Instruct --engine trt_llm --disable_token_auth
---
### 🚢 Quick Start LLM Deployment with Docker
```bash
#export token=<HUGGINGFACE_HUB_TOKEN>
docker build --pull . -f docker/Dockerfile.vllm -t ts/vllm
docker run --rm -ti --shm-size 10g --gpus all -e HUGGING_FACE_HUB_TOKEN=$token -p 8080:8080 -v data:/data ts/vllm --model_id meta-llama/Meta-Llama-3-8B-Instruct --disable_token_auth
For agents
This page has a .md twin and JSON over the API.