Why training on Multiple GPU is slower than training on Single GPU for fine tuning Speech to Text Model
### System Info

- `transformers` version: 4.24.0.dev0
- Platform: Linux-5.15.0-52-generic-x86_64-with-debian-bookworm-sid
- Python version: 3.7.13
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.10.0+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help?

Speech: @patrickvonplaten,  @anton-l, @sanchit-gandhi

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

For training Wav2Vec2 model on multiple gpus, small changes I have made in the script  - `run_speech_recognition_ctc.py` for loading custom hindi dataset and no further changes made. Just modified parameter: `nproc_per_node` for number of gpus in run.sh:
```
OMP_NUM_THREADS=1 python -m torch.distributed.launch \
	--nproc_per_node 3 run_speech_recognition_ctc.py \
        --dataset_name="common_voice" \
	--model_name_or_path="facebook/wav2vec2-xls-r-300m" \
	--output_dir="./new_output" \
	--overwrite_output_dir \
	--num_train_epochs="30" \
	--per_device_train_batch_size="8" \
	--per_device_eval_batch_size="8" \
	--learning_rate="3e-4" \
	--warmup_steps="500" \
	--evaluation_strategy="steps" \
	--text_column_name="sentence" \
	--length_column_name="input_length" \
	--save_steps="200" \
	--eval_steps="200" \
	--logging_steps="200" \
	--layerdrop="0.0" \
	--save_total_limit="2" \
	--freeze_feature_encoder \
	--gradient_checkpointing \
	--chars_to_ignore \। \| \’ \– \, \? \. \! \- \; \: \" \“ \% \‘ \” \� \' \
	--fp16 \
	--group_by_length \
	--do_train --do_eval
```
Original code provided in this [repository](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition#common-voice-ctc).

For training on single gpu, small changes were made for loading custom dataset. Code used from this blog: https://huggingface.co/blog/fine-tune-xlsr-wav2vec2. Rest of the code is same. 

### Expected behavior

Using one GPU:
![image](https://user-images.githubusercontent.com/110966055/198202949-64dac0b5-3693-4941-bc71-73086563999a.png)

At 4 mins, already reached 200 epochs.

Using Multiple GPU:
![image](https://user-images.githubusercontent.com/110966055/198203147-d4a8a5a2-8b8c-49d8-a0bd-1aba5f3bf0c1.png)

At 4 mins, only reached 80 epochs.

Using multiple gpu should speed up the training/fine-tuning process. But instead, it is slower. Kindly need your support to check this issue.
@sanchit-gandhi could you take a look here?<|||||>Hey @ishamnewsreels! A couple of things:
- Do we need to set `OMP_NUM_THREADS=1`? Looks like this affects multi-processing, wondering if it's interfering with distributed training
- Whilst distributed training is running, could you open a new command line window and execute the Unix command:
```bash
watch -n 0.1 nvidia-smi
```
This will launch the NVIDIA system management interface, and display individual GPU usage. We'd expect all three of your GPUs to be in use for distributed training. If < 3 are being used there's an issue with launching distributed training!<|||||>Hi @sanchit-gandhi. 

If I do not set `OMP_NUM_THREADS=1`, the code doesn't execute at all. 
Also, I have used the unix command that you mentioned and I can observe that all 3 gpus are used. 
![image](https://user-images.githubusercontent.com/110966055/199231670-07aea7d3-3488-4b62-9f4e-361683e6f712.png)
<|||||>Hey @ishamnewsreels - that's good to see that all three GPUs are used. I think I see what the problem is! You have the number of epochs fixed as 30, but are changing the effective batch size for single vs multi GPU training. This changes the number of optimisation steps (= num epochs * epoch-size / batch-size).

With single GPU, your settings were as follows:
- `per_device_batch_size` = 8
- `gradient_accumulation_steps` = 2
- Number of devices = 1
- Effective batch size = `per_device_batch_size` * `gradient_accumulation_steps` * number of devices = 16
- For 30 epochs, this gives **11940** optimisation steps

With three GPUs, your settings were as follows:
- `per_device_batch_size` = 8
- `gradient_accumulation_steps` = 1
- Number of devices = 3
- Effective batch size = `per_device_batch_size` * `gradient_accumulation_steps` * number of devices = 24 (1.5x more what we had for single GPU)
- For 30 epochs, this gives **7980** optimisation steps (1.5x less than what we had for single GPU)

The progress bars that we see during training are **not** the number of epochs, but rather the number of **optimisation steps**. With multi-GPU, we're training for fewer optimisation steps (as the batch size is larger), and so we expect the number of optimisation steps to be less after 4 minutes. After 4 minutes, the % of training completed is 1.67% for single GPU, and 1.00% for multi GPU -> so the training progress is quite similar after this time. We can attribute the difference in training progress to the added communication cost in using multi GPU vs single GPU (we have to sync the GPU's up when we do multi GPU training, giving a communication overhead).

Hope that makes sense!