{"data":{"slug":"hpcaitech-swiftinfer","name":"SwiftInfer","tagline":"Efficient AI Inference Serving","github_url":"https://github.com/hpcaitech/SwiftInfer","owner":"hpcaitech","repo":"SwiftInfer","owner_avatar_url":"https://avatars.githubusercontent.com/u/88699314?v=4","primary_language":"Python","stars":476,"forks":31,"topics":["artificial-intelligence","deep-learning","gpt","inference","llama","llama2","llm-inference","llm-serving"],"archived":false,"github_pushed_at":"2024-01-08T09:18:42+00:00","maintenance_label":"Dormant","stars_delta_30d":-2,"url":"https://www.graphcanon.com/tools/hpcaitech-swiftinfer","markdown_url":"https://www.graphcanon.com/tools/hpcaitech-swiftinfer.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/hpcaitech-swiftinfer","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=hpcaitech-swiftinfer","description":"Efficient AI Inference & Serving","homepage_url":"https://hpc-ai.com/","license":"Apache-2.0","open_issues":3,"watchers":4,"ai_summary":"A Python-based repository for efficient inference and serving of artificial intelligence models, specializing in deep-learning frameworks including GPT, LLaMA, and LLaMA2.","readme_excerpt":"### 🛠 Installation\n\nWe use the API in [**TensorRT-LLM**](https://github.com/NVIDIA/TensorRT-LLM) to construct the model and run inference. As the API of TensorRT-LLM is not stable and changing rapidly, we bind our implementation with the `42af740db51d6f11442fd5509ef745a4c043ce51` commit whose version is `v0.6.0`. We may upgrade this repository as TensorRT-LLM's APIs become more stable.\n\nIf you have build **TensorRT-LLM V0.6.0**, simply run:\n\n```bash\ngit clone https://github.com/hpcaitech/SwiftInfer.git\ncd SwiftInfer\npip install .\n```\n\nOtherwise, you should install TensorRT-LLM first.\n\n#### Install TensorRT-LLM with Docker\n\nIf using docker, you can follow [TensorRT-LLM Installation](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/installation.md) to install **TensorRT-LLM V0.6.0**.\n\nBy using docker, you can install SwiftInfer by simply running:\n\n```bash\ngit clone https://github.com/hpcaitech/SwiftInfer.git\ncd SwiftInfer\npip install .\n```\n\n#### Install TensorRT-LLM without Docker\n\nIf not using docker, we provide a script to install TensorRT-LLM automatically.\n\n**Prerequisites**\n\nPlease ensure that you have installed the following packages:\n\n- python\n- build essentials, including gcc/g++, make, cmake\n- CUDA toolkit\n- cuDNN\n- NCCL\n- TensorRT\n- PyTorch\n\nMake sure the version of TensorRT >= 9.1.0 and CUDA toolkit >= 12.2.\n\nTo install tensorrt:\n\n```bash\nARCH=$(uname -m)\nif [ \"$ARCH\" = \"arm64\" ];then ARCH=\"aarch64\";fi\nif [ \"$ARCH\" = \"amd64\" ];then ARCH=\"x86_64\";fi\nif [ \"$ARCH\" = \"aarch64\" ];then OS=\"ubuntu-22.04\"; else OS=\"linux\";fi\nwget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/9.1.0/tars/tensorrt-9.1.0.4.$OS.$ARCH-gnu.cuda-12.2.tar.gz\ntar xzvf tensorrt-9.1.0.4.linux.x86_64-gnu.cuda-12.2.tar.gz\nPY_VERSION=$(python -c 'import sys; print(\".\".join(map(str, sys.version_info[0:2])))')\nPARSED_PY_VERSION=$(echo \"${PY_VERSION//./}\")\npip install TensorRT-9.1.0.4/python/tensorrt-*-cp${PARSED_PY_VERSION}-*.whl\nexport TRT_ROOT=$(realpath TensorRT-9.1.0.4)\n```\n\nTo download nccl, follow [NCCL download page](https://developer.nvidia.com/nccl/nccl-download).\n\nTo download cudnn, follow [cuDNN download page](https://developer.nvidia.com/rdp/cudnn-download).\n\n**Commands**\n\nBefore running the following commands, please ensure that you have set `nvcc` correctly. To check it, run:\n\n```bash\nnvcc --version\n```\n\nTo install TensorRT-LLM and SwiftInfer, run:\n\n```bash\ngit clone https://github.com/hpcaitech/SwiftInfer.git\ncd SwiftInfer\nTRT_ROOT=xxx NCCL_ROOT=xxx CUDNN_ROOT=xxx pip install .\n```","github_created_at":"2024-01-07T15:03:40+00:00","created_at":"2026-07-11T11:45:58.59313+00:00","updated_at":"2026-08-25T12:00:55.541359+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"artificial-intelligence","name":"artificial-intelligence"},{"slug":"deep-learning","name":"deep-learning"},{"slug":"gpt","name":"gpt"},{"slug":"inference","name":"inference"},{"slug":"llama","name":"llama"},{"slug":"llm-inference","name":"llm-inference"},{"slug":"llm-serving","name":"llm-serving"},{"slug":"llma2","name":"llma2"}],"trust":{"provenance":{"is_fork":false,"github_id":740111119,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-25T12:00:54.709Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":960,"last_release_at":null,"stars_delta_30d":-2,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:46:00.035Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-25T12:00:55.167Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-25T12:00:55.167Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-25T12:00:55.167Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need to efficiently serve models from popular frameworks like GPT, LLaMA, or LLaMA2 within a Python environment.","If your project benefits from an open-source solution licensed under Apache-2.0."],"when_not_to_use":["Avoid if your primary model framework is not supported by SwiftInfer, such as TensorFlow or other non-listed frameworks.","Do not use if you require a language other than Python for inference serving."],"source":"enrich:decision_facts","observed_at":"2026-07-15T09:04:51.821Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"SwiftInfer specializes in efficient inference and serving of deep-learning models including GPT, LLaMA, and LLaMA2."}]}}