Language modeling example script missing the next sentence predicion
The example script in `run_language_modeling.py` does not include the next sentence prediction for pre-training BERT. I was wondering if that is a) an oversight, b) for simplicity, or c) because you have found its impact to be non-significant? 
It is both b) and c) :).<|||||>Hello @LysandreJik  and @BramVanroy 

Did you have any results in training   run_language_modeling.py    for some language from scratch (i mean with and  without NSP (next sentence prediction) as is in that script) ?

Did you get better or relatively the same losses (and perplexity, accuracy) by first doing MLM then 
NSP ?

what were your warmup steps and  block size (maximum sequence length) for each (MLM , NSP) task , if you have down them separately ? <|||||>as @LysandreJik mentioned here
 
https://github.com/huggingface/transformers/issues/2693#issuecomment-589819382

"the RoBERTa paper has proven that the NSP objective was not particularly helpful"
 
Is that also right for BERT ?  (training for some language from scratch)
<|||||>If it is right  ,  I think we can set  block size (maximum sequence length)  equal to 64  for MLM
 
Because the  original paper used  128 (except for 10% of the last steps)  , and it had two sentences (A and B for the sake of NSP and model learns that it shouldn't see B for filling masked words in A, because A and B aren't relevant to each other half of the times) which means average block size of 64 for each one 

And it means grate speed up in training BERT, if I say right
because, I haven't got a TPU or even enough GPU to do that 

As mentioned in original paper of BERT "Longer sequences are disproportionately expensive because attention is quadratic to the sequence length"<|||||>Perhaps, another idea is that 
words in sentence A do attention on sentence B, anyway
and that attention is very important to get great results in MLM task (here I mean block size of 128 which means 64 for A and another 64 for B)

even regarding the fact that  A and B are related sentences, just half of the times 
(and that attentions are the main inputs for task NSP)

And by using block size of 64 (and not doing task NSP during the training) , I will get very bad results 