BERT Trainer.train() CUDA out of memory error
## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-5.3.0
- Python version: 3.6.10
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Distributed

### Who can help
@LysandreJik, @sgugger, @patrickvonplaten

## Information
I am using 8xV100s (32GB). The script (`run_training.py`) works when running on a single machine but I am running into the ```CUDA out of memory``` when trying to run distributed training. The behavior is consistent whether or not `fp16` is `True`. I am using the publicly available wikitext data. 

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. My code is `run_training.py`:
```
bert_config = BertConfig(hidden_size=768, num_attention_heads=12)
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForPreTraining(config=bert_config)
train_dataset = TextDatasetForNextSentencePrediction(tokenizer=tokenizer, file_path=args.train_data_file, block_size=args.max_length)
data_collator = DataCollatorForNextSentencePrediction(tokenizer=tokenizer, mlm_probability=args.mlm_probability)
trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset
    )
global_steps, training_loss = trainer.train()
```

2. Run the script in distributed mode:
```
python -m torch.distributed.launch \
  --nnodes={num_nodes} \
  --node_rank={rank} \
  --nproc_per_node=8  \
  --run_training.py \
  --max_steps 50 " \
  --train_data_file {train_data_file} " \
  --eval_data_file {test_data_file} " \
  --logging_dir {output_dir} \
  --fp16 False " \
  --gradient_accumulation_steps 1 " \
  --per_gpu_train_batch_size 1 \
  --per_gpu_eval_batch_size 1
```

## Expected behavior

Given that the script works fine (i.e., not run into the out of memory issue) on a single machine, I would expect multi-node to be the same. Any insight into what might be going on is appreciated!
Haven’t we fixed a memory leak on master recently?<|||||>Yes we did, though this might be a different problem.

To make sure, can you please check if you have the bug with an install from source?<|||||>@sgugger yeah, I'm installing from source (not sure why the script yielded 3.0.2)<|||||>I encountered the problem, too, especially using NSP objective, the memory usage much higher than MLM.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I encounter this problem too when I try to train on multi-nodes and multi GPUs. 
And my transformers version is 4.15.0, so does this issue have the answer?
@choidongyeon @julien-c @sgugger 