Hi @rgwatwormhill, if I read the documentation from huggingface ,
BertForPreTraining provides two training heads mlm + nsp.(https://huggingface.co/transformers/model_doc/bert.html#transformers.BertForPreTraining 144)
Also according to this PR on github (https://github.com/huggingface/transformers/pull/6644 28), the DatasetforNSP and DatacollatorforNSP, both of them take care for MLM +NSP data formatiing.  I expected using both (DatasetforNSP and DatacollatorforNSP) with BertForPreTraining should be fine but I may be wrong. Also for the batch size you are correct, it may be too large for the training, however, the error which I am getting has some other problem, and does not successfully go to the training.