 Model Description  This model was captured during a reproduction of [BERT-base uncased](https://github.com/google-research/bert), for English: it is a Transformers model pretrained on a large corpus of English data, using the Masked Language Modelling (MLM) and the Next Sentence Prediction (NSP) objectives.  The intended uses, limitations, training data and training procedure for the fully trained model are similar to [BERT-base uncased](https://github.com/google-research/bert). Two major differences with the original model:  *   We pre-trained the MultiBERTs models for 2 million steps using sequence     length 512 (instead of 1 million steps using sequence length 128 then 512). *   We used an alternative version of Wikipedia and Books Corpus, initially     collected for [Turc et al., 2019](https://arxiv.org/abs/1908.08962).  This is a best-effort reproduction, and so it is probable that differences with the original model have gone unnoticed. The performance of MultiBERTs on GLUE after full training is oftentimes comparable to that of original BERT, but we found significant differences on the dev set of SQuAD (MultiBERTs outperforms original BERT). See our [technical report](https://arxiv.org/abs/2106.16163) for more details.  