The library provides a version of the model for masked language modeling, token classification, sentence
classification, multiple choice classification and question answering.
FlauBERT
FlauBERT: Unsupervised Language Model Pre-training for French, Hang Le et al.
Like RoBERTa, without the sentence ordering prediction (so just trained on the MLM objective).
The library provides a version of the model for language modeling and sentence classification.
ELECTRA
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators,
Kevin Clark et al.
ELECTRA is a transformer model pretrained with the use of another (small) masked language model. The inputs are
corrupted by that language model, which takes an input text that is randomly masked and outputs a text in which ELECTRA
has to predict which token is an original and which one has been replaced. Like for GAN training, the small language
