This is a summary of the models available in 🤗 Transformers. It assumes you’re familiar with the original transformer
model. For a gentle introduction check the annotated transformer. Here we focus on the high-level differences between the
models. You can check them more in detail in their respective documentation. Also check out the Model Hub where you can filter the checkpoints by model architecture.
Each one of the models in the library falls into one of the following categories:
autoregressive-models
autoencoding-models
seq-to-seq-models
multimodal-models
retrieval-based-models
Autoregressive models are pretrained on the classic language modeling task: guess the next token having read all the
previous ones. They correspond to the decoder of the original transformer model, and a mask is used on top of the full
