learning. We equip BERT with this mixed attention design and build a ConvBERT model. Experiments have shown that
ConvBERT significantly outperforms BERT and its variants in various downstream tasks, with lower training cost and
fewer model parameters. Remarkably, ConvBERTbase model achieves 86.4 GLUE score, 0.7 higher than ELECTRAbase, while
using less than 1/4 training cost.
The library provides a version of the model for masked language modeling, token classification, sentence classification
and question answering.
XLM
Cross-lingual Language Model Pretraining, Guillaume Lample and Alexis Conneau
A transformer model trained on several languages. There are three different type of training for this model and the
library provides checkpoints for all of them:
Causal language modeling (CLM) which is the traditional autoregressive training (so this model could be in the
