GraphCanon updated 3w · GitHub synced 3w
Decision brief
SwiftInfer specializes in efficient inference and serving of deep-learning models including GPT, LLaMA, and LLaMA2.
Good fit when
- When you need to efficiently serve models from popular frameworks like GPT, LLaMA, or LLaMA2 within a Python environment.
- If your project benefits from an open-source solution licensed under Apache-2.0.
Avoid when
- Avoid if your primary model framework is not supported by SwiftInfer, such as TensorFlow or other non-listed frameworks.
- Do not use if you require a language other than Python for inference serving.
Observed Jul 15, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (929d since push)
- As of 3w
- Provenance
- Not a fork · Organization account
- As of 3w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
pip install SwiftInferSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
A Python-based repository for efficient inference and serving of artificial intelligence models, specializing in deep-learning frameworks including GPT, LLaMA, and LLaMA2.
Capability facts
- Languages
- python
Source: github.language · Jul 25, 2026
Categories
Compatibility
Sourced claims from the README excerpt - not unsourced marketing copy.
Tags
README
🛠 Installation
We use the API in TensorRT-LLM to construct the model and run inference. As the API of TensorRT-LLM is not stable and changing rapidly, we bind our implementation with the 42af740db51d6f11442fd5509ef745a4c043ce51 commit whose version is v0.6.0. We may upgrade this repository as TensorRT-LLM's APIs become more stable.
If you have build TensorRT-LLM V0.6.0, simply run:
git clone https://github.com/hpcaitech/SwiftInfer.git
cd SwiftInfer
pip install .
Otherwise, you should install TensorRT-LLM first.
Install TensorRT-LLM with Docker
If using docker, you can follow TensorRT-LLM Installation to install TensorRT-LLM V0.6.0.
By using docker, you can install SwiftInfer by simply running:
git clone https://github.com/hpcaitech/SwiftInfer.git
cd SwiftInfer
pip install .
Install TensorRT-LLM without Docker
If not using docker, we provide a script to install TensorRT-LLM automatically.
Prerequisites
Please ensure that you have installed the following packages:
- python
- build essentials, including gcc/g++, make, cmake
- CUDA toolkit
- cuDNN
- NCCL
- TensorRT
- PyTorch
Make sure the version of TensorRT >= 9.1.0 and CUDA toolkit >= 12.2.
To install tensorrt:
ARCH=$(uname -m)
if [ "$ARCH" = "arm64" ];then ARCH="aarch64";fi
if [ "$ARCH" = "amd64" ];then ARCH="x86_64";fi
if [ "$ARCH" = "aarch64" ];then OS="ubuntu-22.04"; else OS="linux";fi
wget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/9.1.0/tars/tensorrt-9.1.0.4.$OS.$ARCH-gnu.cuda-12.2.tar.gz
tar xzvf tensorrt-9.1.0.4.linux.x86_64-gnu.cuda-12.2.tar.gz
PY_VERSION=$(python -c 'import sys; print(".".join(map(str, sys.version_info[0:2])))')
PARSED_PY_VERSION=$(echo "${PY_VERSION//./}")
pip install TensorRT-9.1.0.4/python/tensorrt-*-cp${PARSED_PY_VERSION}-*.whl
export TRT_ROOT=$(realpath TensorRT-9.1.0.4)
To download nccl, follow NCCL download page.
To download cudnn, follow cuDNN download page.
Commands
Before running the following commands, please ensure that you have set nvcc correctly. To check it, run:
nvcc --version
To install TensorRT-LLM and SwiftInfer, run:
git clone https://github.com/hpcaitech/SwiftInfer.git
cd SwiftInfer
TRT_ROOT=xxx NCCL_ROOT=xxx CUDNN_ROOT=xxx pip install .
For agents
This page has a .md twin and JSON over the API.