{"data":{"slug":"nvidia-fastertransformer","name":"FasterTransformer","tagline":"Transformer related optimization including BERT and GPT","github_url":"https://github.com/NVIDIA/FasterTransformer","owner":"NVIDIA","repo":"FasterTransformer","owner_avatar_url":"https://avatars.githubusercontent.com/u/1728152?v=4","primary_language":"C++","stars":6448,"forks":934,"topics":["bert","gpt","pytorch","transformer"],"archived":false,"github_pushed_at":"2024-03-27T11:25:30+00:00","maintenance_label":"Dormant","stars_delta_30d":2,"url":"https://www.graphcanon.com/tools/nvidia-fastertransformer","markdown_url":"https://www.graphcanon.com/tools/nvidia-fastertransformer.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/nvidia-fastertransformer","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=nvidia-fastertransformer","description":"Transformer related optimization, including BERT, GPT","homepage_url":null,"license":"Apache-2.0","open_issues":289,"watchers":59,"ai_summary":"Repository for NVIDIA's FasterTransformer, focusing on highly optimized transformer-based encoder and decoder components for inference. Notably supports BERT and GPT models with integration options for TensorFlow, PyTorch, and Triton backend.","readme_excerpt":"**Note: FasterTransformer development has transitioned to [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM/tree/release/0.5.0). All developers are encouraged to leverage TensorRT-LLM to get the latest improvements on LLM Inference. The NVIDIA/FasterTransformer repo will stay up, but will not have further development.**\n\n# FasterTransformer\n\nThis repository provides a script and recipe to run the highly optimized transformer-based encoder and decoder component, and it is tested and maintained by NVIDIA.\n\n## Table Of Contents\n\n- [FasterTransformer](#fastertransformer)\n  - [Table Of Contents](#table-of-contents)\n  - [Model overview](#model-overview)\n    - [Support matrix](#support-matrix)\n  - [Advanced](#advanced)\n    - [Global Environment](#global-environment)\n  - [Performance](#performance)\n    - [BERT base performance](#bert-base-performance)\n      - [BERT base performances of FasterTransformer new features](#bert-base-performances-of-fastertransformer-new-features)\n      - [BERT base performance on TensorFlow](#bert-base-performance-on-tensorflow)\n      - [BERT base performance on PyTorch](#bert-base-performance-on-pytorch)\n    - [Decoding and Decoder performance](#decoding-and-decoder-performance)\n      - [Decoder and Decoding end-to-end translation performance on TensorFlow](#decoder-and-decoding-end-to-end-translation-performance-on-tensorflow)\n      - [Decoder and Decoding end-to-end translation performance on PyTorch](#decoder-and-decoding-end-to-end-translation-performance-on-pytorch)\n    - [GPT performance](#gpt-performance)\n  - [Release notes](#release-notes)\n    - [Changelog](#changelog)\n    - [Known issues](#known-issues)\n\n## Model overview\n\nIn NLP, encoder and decoder are two important components, with the transformer layer becoming a popular architecture for both components. FasterTransformer implements a highly optimized transformer layer for both the encoder and decoder for inference. On Volta, Turing and Ampere GPUs, the computing power of Tensor Cores are used automatically when the precision of the data and weights are FP16.\n\nFasterTransformer is built on top of CUDA, cuBLAS, cuBLASLt and C++. We provide at least one API of the following frameworks: TensorFlow, PyTorch and Triton backend. Users can integrate FasterTransformer into these frameworks directly. For supporting frameworks, we also provide example codes to demonstrate how to use, and show the performance on these frameworks.\n\n### Support matrix\n\n| Models           | Framework      | FP16 | INT8 (after Turing) | Sparsity (after Ampere) | Tensor parallel | Pipeline parallel | FP8 (after Hopper) |\n| ---------------- | -------------- | ---- | ------------------- | ----------------------- | --------------- | ----------------- | ------------------ |\n| BERT             | TensorFlow     | Yes  | Yes                 | -                       | -               | -                 | -                  |\n| BERT             | PyTorch        | Yes  | Yes                 | Yes                     | Yes             | Yes               | -                  |\n| BERT             | Triton backend | Yes  | -                   | -                       | Yes             | Yes               | -                  |\n| BERT             | C++            | Yes  | Yes                 | -                       | -               | -                 | Yes                |\n| XLNet            | C++            | Yes  | -                   | -                       | -               | -                 | -                  |\n| Encoder          | TensorFlow     | Yes  | Yes                 | -                       | -               | -                 | -                  |\n| Encoder          | PyTorch        | Yes  | Yes                 | Yes                     | -               | -                 | -                  |\n| Decoder          | TensorFlow     | Yes  | -                   | -                       | -               | -                 | -                  |\n| Decode","github_created_at":"2021-04-02T21:36:33+00:00","created_at":"2026-07-11T10:37:01.467536+00:00","updated_at":"2026-09-20T04:23:24.661198+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"bert","name":"bert"},{"slug":"cublas","name":"cublas"},{"slug":"cublaslt","name":"cublaslt"},{"slug":"cuda","name":"cuda"},{"slug":"gpt","name":"gpt"},{"slug":"inference-optimization","name":"inference optimization"},{"slug":"pytorch","name":"pytorch"},{"slug":"transformer","name":"transformer"}],"trust":{"provenance":{"is_fork":false,"github_id":354138793,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-07T00:01:08.563Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":893,"last_release_at":"2023-01-23T13:01:46Z","stars_delta_30d":2,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:37:03.035Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-07T00:01:09.073Z"},"languages":{"value":["c++"],"source":"github.language","observed_at":"2026-09-07T00:01:09.073Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-07T00:01:09.073Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When aiming for high performance with GPU-based FP16 computations for BERT or GPT models specifically.","For integrating highly optimized API into TensorFlow, PyTorch, or Triton backend."],"when_not_to_use":["If looking for active development and latest improvements on LLM Inference as NVIDIA recommends TensorRT-LLM over FasterTransformer now.","When specific frameworks not including TensorFlow, PyTorch, or Triton are required."],"source":"enrich:decision_facts","observed_at":"2026-07-12T17:25:18.297Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Highly optimized transformer encoder and decoder for inferencing, supporting BERT and GPT on various frameworks like TensorFlow, PyTorch."}]}}