{"data":{"slug":"pytorch-serve","name":"serve","tagline":"Serve, optimize and scale PyTorch models in production","github_url":"https://github.com/pytorch/serve","owner":"pytorch","repo":"serve","owner_avatar_url":"https://avatars.githubusercontent.com/u/21003710?v=4","primary_language":"Java","stars":4344,"forks":880,"topics":["cpu","deep-learning","docker","gpu","kubernetes","machine-learning","metrics","mlops","optimization","pytorch","serving"],"archived":true,"github_pushed_at":"2025-08-06T19:17:08+00:00","maintenance_label":"Archived","stars_delta_30d":-6,"url":"https://www.graphcanon.com/tools/pytorch-serve","markdown_url":"https://www.graphcanon.com/tools/pytorch-serve.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/pytorch-serve","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=pytorch-serve","description":"Serve, optimize and scale PyTorch models in production","homepage_url":"https://pytorch.org/serve/","license":"Apache-2.0","open_issues":443,"watchers":11,"ai_summary":"A toolkit for deploying PyTorch models in production environments, supporting optimization and scaling with deep learning frameworks and Kubernetes.","readme_excerpt":"### 🚀 Quick start with TorchServe\n\n```bash\n\n---\n\n# Install dependencies\npython ./ts_scripts/install_dependencies.py\n\n---\n\n### 🚀 Quick start with TorchServe (conda)\n\n```bash\n\n---\n\n### 🤖 Quick Start LLM Deployment\n\n#### VLLM Engine\n```bash\n\n---\n\n# Make sure to install torchserve with pip or conda as described above and login with `huggingface-cli login`\npython -m ts.llm_launcher --model_id meta-llama/Llama-3.2-3B-Instruct --disable_token_auth\n\n---\n\n# Make sure to install torchserve with python venv as described above and login with `huggingface-cli login`\n\n---\n\n# pip install -U --use-deprecated=legacy-resolver -r requirements/trt_llm.txt\npython -m ts.llm_launcher --model_id meta-llama/Meta-Llama-3.1-8B-Instruct --engine trt_llm --disable_token_auth\n\n---\n\n### 🚢 Quick Start LLM Deployment with Docker\n\n```bash\n#export token=<HUGGINGFACE_HUB_TOKEN>\ndocker build --pull . -f docker/Dockerfile.vllm -t ts/vllm\n\ndocker run --rm -ti --shm-size 10g --gpus all -e HUGGING_FACE_HUB_TOKEN=$token -p 8080:8080 -v data:/data ts/vllm --model_id meta-llama/Meta-Llama-3-8B-Instruct --disable_token_auth","github_created_at":"2019-10-03T03:17:43+00:00","created_at":"2026-07-11T23:12:45.393703+00:00","updated_at":"2026-09-20T04:29:59.370183+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"cpu","name":"cpu"},{"slug":"deep-learning","name":"deep-learning"},{"slug":"docker","name":"docker"},{"slug":"gpu","name":"gpu"},{"slug":"kubernetes","name":"kubernetes"},{"slug":"machine-learning","name":"machine-learning"},{"slug":"metrics","name":"metrics"},{"slug":"mlops","name":"mlops"}],"trust":{"provenance":{"is_fork":false,"github_id":212488700,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:20:12.433Z","maintenance":{"label":"Archived","score":8,"methodology":"github_public_v1","releases_90d":0,"days_since_push":407,"last_release_at":"2024-09-30T22:46:39Z","stars_delta_30d":-6,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:20:15.035Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:20:11.303Z"},"languages":{"value":["java"],"source":"github.language","observed_at":"2026-09-18T08:20:11.303Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:20:11.303Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need to deploy PyTorch models and require support for optimization and scaling with deep learning frameworks and Kubernetes.","If your project involves LLM deployment and you want to use the VLLM engine or TRT-LLM engine for specific models like Llama-3.2-3B-Instruct or Meta-Llama-3.1-8B-Instruct.","When you prefer deployment options that include TorchServe, Docker, and conda environments, and you are working with Java and Apache-2.0 licensed projects."],"when_not_to_use":["If your project does not involve PyTorch models or if you do not require deep learning framework and Kubernetes support for scaling and optimization.","If you are working with models that are not supported by the VLLM engine or TRT-LLM engine, such as models that are not listed in the repository's quick start examples.","If you are not working with Java or do not want to use an Apache-2.0 licensed toolkit, or if you prefer a different deployment method not covered by Serve, such as a different containerization or ML-ö"],"source":"enrich:decision_facts","observed_at":"2026-09-18T10:40:13.194Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Serve is a toolkit for deploying PyTorch models in production, focusing on optimization and scaling with deep learning frameworks and Kubernetes. It supports deployment via TorchServe, Docker, and conda environments."}]}}