Slow Multi-GPU DDP training with run_clm.py and GPT2
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0
- Platform: Linux-3.10.0-1160.11.1.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.6
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes (using official `run_clm.py`)
- Using distributed or parallel set-up in script?: using DDP with `run_clm.py`

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
- gpt2: @patrickvonplaten, @LysandreJik
- trainer, maintained examples: @sgugger

## Information

Model I am using (Bert, XLNet ...): gpt2-medium

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Generate some dummy text data (3000 examples) and save a csv:
```python
import pandas as pd
text = " ".join(100*["Here is some very long text."])
text = 3000*[text]
pd.Series(text).to_frame("text").to_csv("data_temp.csv",index=False)
```
2. Run official [`run_clm.py`](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py) from examples using 3 gpus and DDP:
```shell
CUDA_VISIBLE_DEVICES="1,2,3" python -m torch.distributed.launch --nproc_per_node 3 run_clm.py \
--model_name_or_path gpt2-medium \
--do_train \
--output_dir /proj/semafor/kirill/tmp \
--per_device_train_batch_size 4 \
--block_size 128 \
--train_file data_temp.csv \
--fp16
```
3. Using 3 GeForce RTX 2080Ti with 11Gbs, tqdm says it should approximately take 1 hour: `48/4101 [00:38<53:51,  1.25it/s`. The memory in each GPU is maxed out: ` 10782MiB / 11019MiB `

4. Now, if I just run the same script on a single GPU:
```shell
CUDA_VISIBLE_DEVICES="3" python run_clm.py \
--model_name_or_path gpt2-medium \
--do_train \
--output_dir /proj/semafor/kirill/tmp \
--per_device_train_batch_size 4 \
--block_size 128 \
--train_file data_temp.csv \
--fp16
```

It's actually a little faster: `260/12303 [00:57<44:02,  4.56it/s`  and the GPU memory is not maxed out: `9448MiB / 11019MiB`

I can actually double the `--per_device_train_batch_size` from `4 -> 8` and get it down to under 30 mins per epoch: `138/6153 [00:36<26:30,  3.78it/s`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

So I expected that:
-  DDP training on 3 GPUs to be faster than a single GPU (It's actually a little slower).
-  If I can load a batch of 8 on a device in a single GPU mode then it should work in multi-GPU mode as well (it doesn't, I get OOM error).
The answers to those two points is not necessarily yes.
- DDP training is only faster if you have NVLinks between your GPUs, otherwise the slow communication between them can slow down training.
- DDP training takes more space on GPU then a single-process training since there is some gradients caching.

Both issues come from PyTorch and not us, the only thing we can check on our side is if there is something in our script that would introduce a CPU-bottleneck, but I doubt this is the reason here (all tokenization happens before the training so there is nothing I could think of there).

You should also try regular `DataParalell` which does not have the memory problem IIRC, but I don't remember the comparison in terms of speed. I think @stas00 may have more insight there.<|||||>Thanks. I’ll give DP a try.<|||||>What @sgugger said, 

please see this excellent benchmark: https://github.com/huggingface/transformers/issues/9371#issuecomment-768656711 You can see that a single GPU beats DDP over 2 gpus there if it's not NVLink-connected. I haven't tried it on 3 gpus though. Surely it should be somewhat faster at least. 

Did you check you're feeding the gpus fast enough? - i.e. check their utilization %, it they are under 90% then you probably have an issue with loading - add more dataloader workers.

Also please consider using DeepSpeed ZeRO-DP, which should be even faster. https://huggingface.co/blog/zero-deepspeed-fairscale<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.