Multi-GPU seq2seq example evaluation significantly slower than legacy example evaluation

### Who can help
@patil-suraj @sgugger 

Models:
T5

## Information

I've been doing multi-GPU evaluation for some weeks using a Transformers pull from Feb 12th, just using the example scripts for training/evaluating custom datasets (specifically `run_distributed_eval.py` , though that seq2seq example is now legacy: https://github.com/huggingface/transformers/tree/master/examples/legacy/seq2seq ) 

Today I grabbed a fresh pull and migrated the data over to the JSON lines format for the new seq2seq example `run_summarization.py` : https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_summarization.py

run_summarization.py appears to use all visible GPUs to do the evaluation (great!), but it also appears significantly slower than the old run_distributed_eval.py .

When examining GPU utilization using `nvtop`, it appears that it allocates GPU memory from all devices (much more from device 0), but only uses device 0 for processing:

![image](https://user-images.githubusercontent.com/3813268/113463500-dde72a00-93da-11eb-9f60-f0b52b182cee.png)

## Script

In case it's my issue and I'm not invoking it correctly (I know the legacy one required being invoked with `torch.distributed.launch` for multi-GPU evaluation), the runscript I'm using is:
```
#/bin/bash
python run_summarization.py \
    --model_name_or_path mymodel-debug1000 \
    --do_predict \
    --train_file mydata/train.json \
    --validation_file mydata/val.json \
    --test_file mydata/val.json \
    --max_source_length 256 \
    --max_target_length 512 \
    --num_beams 8 \
    --source_prefix "" \
    --output_dir tst-debug \
    --overwrite_output_dir \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --predict_with_generate

```
You should compare apple to apple: if you're not launching the script with `torch.distributed.launch` you're not comparing the same kinds of distributed training (not using the launcher uses `DataParallel` behind the scenes whereas the launcher uses `DistributedDataParallel`).<|||||>Ah, thanks -- it's my issue then.  I didn't realize that you could also invoke the new script with `torch.distributed.launch` since the section on multi-GPU evaluation was removed from the README, and when I saw it using all 4 GPUs I just assumed it was taking care of it all internally now.  Thanks!

It might be helpful to add a small section on multi-GPU evaluation to the example seq2seq README (e.g. here's the text from the now legacy README: https://github.com/huggingface/transformers/tree/master/examples/legacy/seq2seq#readme )? 

<|||||>Yes indeed. Would you mind opening a PR with a suggestion?<|||||>Of course, thanks -- I'm just testing it out before hand.

Any sense of why the memory allocation would look so imbalanced across GPUs with `torch.distributed.launch`?  IIRC it was fairly uniform with the legacy code? 

![image](https://user-images.githubusercontent.com/3813268/113614636-9beefb80-9607-11eb-8178-068eada3577b.png)

FYI this is just a baseline test, with a batch size of 1 (num_beams=32): 
```
export BS=1

PYTHONPATH=../../src python -m torch.distributed.launch --nproc_per_node=4  ./run_summarization.py \
    --model_name_or_path mymodel-debug1000 \
    --do_predict \
    --train_file mydata/train.json \
    --validation_file mydata/val.json \
    --test_file mydata/test.json \
    --max_source_length 256 \
    --max_target_length 512 \
    --num_beams 32 \
    --source_prefix "" \
    --output_dir tst-debug-dist \
    --overwrite_output_dir \
    --per_device_train_batch_size=$BS \
    --per_device_eval_batch_size=$BS \
    --predict_with_generate \
    --fp16 \

```

<|||||>I don't know how the legacy script worked but this one lets each GPU generate up to when every sample in the batch (so in your case the one and only) reach an eos token or the maximum target length. I think it's normal to have some imbalance as a result.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.