Can you write code that trains bert with MLM and the next sentence at the same time?
### Feature request

Or is there a training code already in place?

### Motivation

I've learned that training with both is better than training with mlm alone, by which I mean the generated vector features

### Your contribution

no yet
Hi @xiao12mm, I belive the NSP loss isn't available for model training directly in huggingface. This [thread](https://discuss.huggingface.co/t/continual-pre-training-from-an-initial-checkpoint-with-mlm-and-nsp/6869) provide a script for NSP training.

About this - "training with both is better than training with mlm alone", there has been specific research done to verify the claim. One of the conclusions from "RoBERTa: A Robustly Optimized BERT Pretraining Approach" paper is that - "removing the NSP loss matches or slightly improves downstream task performance". <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.