Multimodal models mix text inputs with other kinds (e.g. images) and are more specific to a given task.
Decoders or autoregressive models
As mentioned before, these models rely on the decoder part of the original transformer and use an attention mask so
that at each position, the model can only look at the tokens before the attention heads.
Original GPT
Improving Language Understanding by Generative Pre-Training, Alec Radford et al.
The first autoregressive model based on the transformer architecture, pretrained on the Book Corpus dataset.
The library provides versions of the model for language modeling and multitask language modeling/multiple choice
classification.
GPT-2
Language Models are Unsupervised Multitask Learners,
Alec Radford et al.
A bigger and better version of GPT, pretrained on WebText (web pages from outgoing links in Reddit with 3 karmas or
