mgreenbe:

BertForPreTraining


The BertForPreTraining model is BERT with 2 heads on top (the ones used for pre-training BERT, namely next sentence prediction and masked language modeling). The bert-base-uncased checkpoint on the hub only includes the language modeling head (it’s actually suited to be loaded into a BertForMaskedLM model). You can also see this in the config file here 5.