Possible bug in "trainer" when training "BertForPretraining.from_pretrained()"
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
Environment is colab with GPU enabled.  modules are provided in the Jupyter notebook on Google Drive here:

[https://colab.research.google.com/drive/1UX6NMXA2cHGUtDJwh_U6LL-kyd8Gyt9y?usp=sharing](https://colab.research.google.com/drive/1UX6NMXA2cHGUtDJwh_U6LL-kyd8Gyt9y?usp=sharing)



### Who can help
@sgugger

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using BERT 
BertForPreTraining.from_pretrained("bert-base-uncased")

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: Error can be reproduced with the Notebook provided above.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)
I am attempting to "fine-tune" bert-base-uncased by training it on additional sentences.  I am unable to do this with Trainer - I get an error message shown in the notebook.

## To reproduce

Steps to reproduce the behavior:

1. Execute the notebook
2. First example succeeds with model BertLMHeadModel.from_pretrained("bert-base-uncased")
3. Second example fails at train() simply by changing the model to BertForPreTraining.from_pretrained("bert-base-uncased")

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

The bug is entirely reproduced on the linked Jupyter Notebook above, when run on Google Colab.  The error message is:

RuntimeError: grad can be implicitly created only for scalar outputs

## Expected behavior
The model "BertForPretraining.from_pretrained("bert-base-uncased") should train on the two sentences provided.

<!-- A clear and concise description of what you would expect to happen. -->

If you know a work-around for this bug, I will appreciate it.

Sylvain - good to see you doing interesting work!! - Dana Ludwig (student of fast.ai course and owner of your book)

Hi Dana! I don't think this is a bug. You are not providing `BertForPreTraining` the `nsp_labels` it also requires for training, so it does not compute the loss (and then the rest fails). You should use `DataCollatorForNextSentencePrediction` to have the batches get those labels too (and it might requires using `TextDatasetForNextSentencePrediction` with it) or write your own `data_collator`  that will add those labels.<|||||>Hi Sylvain!  Thanks for the quick response!  My understanding of the process for pre-training BERT is that it is self-supervised and creates it's own labels.  For example, for "next sentence prediction", it looks at the input sentences and uses the "next sentence" as the label for that task.  That's how it worked when I used the TensorFlow model to train my BERT from scratch.  Does the HuggingFace trainer not do that?  I will look at "extDatasetForNextSentencePrediction" to see if that has some answers.  I just thought that HuggingFace framework would be easier to fine-tune than using Google TensorFlow code.<|||||>The trainer just does the training loop, it is independent from the tasks. Transformers provides tools to get the data together (which I mentioned) and ready for the Trainer on all the most common NLP tasks, BERT-pretraining objective included.<|||||>Hi Sylvain,
Your suggestion did the trick!  I used ‘TextDatasetForNextSentencePrediction’ to build my dataset and ‘DataCollatorForNextSentencePrediction’ for my collator.  It’s training now and the validation loss is getting lower, so everything looks fine.  As you remember, fine-tuning the baseline pre-trained model with new task-specific data was part of your workflow for ULMFIT, so I can imagine this use-case will come up a lot.  If you would like me to clean up my test example notebook, I’d be glad to let you post it in your examples section.  It took me days to get this far, so I’d like to save the next person some time if possible. 

Thank you! Dana