Multi-GPU training has literally no GPU-Utilization (0%)
I know that multi-GPU training is handled by the trainer class automatically through `CUDA_VISIBLE_DEVICES=...` flag in transformers. But, I'm having a weird problem. Like, after setting `CUDA_VISIBLE_DEVICES=0,1,2`, 3 GPUs are being used and `nvidia-smi`outputs the following:

```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:17.0 Off |                    0 |
| N/A   76C    P0   293W / 300W |  13758MiB / 16160MiB |     94%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:00:18.0 Off |                    0 |
| N/A   43C    P0    72W / 300W |   4770MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:00:19.0 Off |                    0 |
| N/A   43C    P0    73W / 300W |   4770MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
```

I'm trying the inference mode of pegasus model:

```
CUDA_VISIBLE_DEVICES=0,1,2 python examples/pytorch/summarization/run_summarization.py \
    --model_name_or_path /home/code-base/user_space/saved_models/pytorch/reddit_tifu/ \
    --do_predict \
    --train_file $DS_BASE_DIR/train.json \
    --validation_file $DS_BASE_DIR/validation.json \
    --test_file $DS_BASE_DIR/test.json \
    --output_dir /home/code-base/user_space/saved_models/pegasus/ \
    --per_device_train_batch_size=3 \
    --per_device_eval_batch_size=3 \
    --overwrite_output_dir \
    --predict_with_generate \
    --text_column text \
    --summary_column summary \
    --num_beams 5
```

The strange thing to me is that GPU-Util of GPU-1 and GPU-2 is 0%, while they got a part of their memory filled. Though, this is not the case about GPU-0. I'm hesitating now if I'm using the correct way of doing multi-GPU training. Any advice or hint would be appreciated!

## Environment info

- `transformers` version: 4.7.0 dev
- Platform: Ubuntu 18.04
- Python version: 3.8
- PyTorch version (GPU?): 1.6
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help
@sgugger

Solved by #11045 

I had to use distributed training to this end. Closing...