Note: FasterTransformer development has transitioned to TensorRT-LLM. All developers are encouraged to leverage TensorRT-LLM to get the latest improvements on LLM Inference. The NVIDIA/FasterTransformer repo will stay up, but will not have further development.

FasterTransformer

This repository provides a script and recipe to run the highly optimized transformer-based encoder and decoder component, and it is tested and maintained by NVIDIA.

Table Of Contents

• FasterTransformer
  • Table Of Contents
  • Model overview
    • Support matrix
  • Advanced
    • Global Environment
  • Performance
    • BERT base performance
      • BERT base performances of FasterTransformer new features
      • BERT base performance on TensorFlow
      • BERT base performance on PyTorch
    • Decoding and Decoder performance
      • Decoder and Decoding end-to-end translation performance on TensorFlow
      • Decoder and Decoding end-to-end translation performance on PyTorch
    • GPT performance
  • Release notes
    • Changelog
    • Known issues

Model overview

In NLP, encoder and decoder are two important components, with the transformer layer becoming a popular architecture for both components. FasterTransformer implements a highly optimized transformer layer for both the encoder and decoder for inference. On Volta, Turing and Ampere GPUs, the computing power of Tensor Cores are used automatically when the precision of the data and weights are FP16.

FasterTransformer is built on top of CUDA, cuBLAS, cuBLASLt and C++. We provide at least one API of the following frameworks: TensorFlow, PyTorch and Triton backend. Users can integrate FasterTransformer into these frameworks directly. For supporting frameworks, we also provide example codes to demonstrate how to use, and show the performance on these frameworks.

Support matrix

Models	Framework	FP16	INT8 (after Turing)	Sparsity (after Ampere)	Tensor parallel	Pipeline parallel	FP8 (after Hopper)
BERT	TensorFlow	Yes	Yes	-	-	-	-
BERT	PyTorch	Yes	Yes	Yes	Yes	Yes	-
BERT	Triton backend	Yes	-	-	Yes	Yes	-
BERT	C++	Yes	Yes	-	-	-	Yes
XLNet	C++	Yes	-	-	-	-	-
Encoder	TensorFlow	Yes	Yes	-	-	-	-
Encoder	PyTorch	Yes	Yes	Yes	-	-	-
Decoder	TensorFlow	Yes	-	-	-	-	-
Decode