how to run pegasus finetune on multiple gpus
## Environment Information
- transformers version: 4.2.0dev0
- Platform: Linux-3.10.0-1062.18.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core
- Python version: 3.7.3
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): 2.4.0 (False)
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

## Who might help
@sgugger
@patrickvonplaten 
@patil-suraj

## Information

The fine-tune process is taking really long time, so I want to do it parallel on multiple gpus.

The problem arises when using:
 I do not have found the instructions for training on multiple gpus for the arguments, are there configurations for something like nodes, etc. or should I implement it in my own script?

## To reproduce
```
python finetune.py \
    --gpus 0 \
    --learning_rate=1e-4 \
    --do_train \
    --do_predict \
    --n_val 1000 \
    --val_check_interval 0.25 \
    --max_source_length 512 --max_target_length 56 \
    --freeze_embeds --label_smoothing 0.1 --adafactor --task summarization_xsum \
    --model_name_or_path google/pegasus-xsum \
    --output_dir=xsum_results \
    --data_dir xsum \
    --tokenizer_name google/pegasus-large \
    "$@"
```
and which of the belowings are correct? I saw both in other posts:

--model_name_or_path google/pegasus-xsum
--tokenizer_name google/pegasus-large \

or
--model_name_or_path google/pegasus-large
--tokenizer_name google/pegasus-xum \

I think it should be the second one but I am not sure. 

## Expected behavior
1. Enable the finetune of pegasus model on multiple gpus.
2. Inject the correct arguments.
Please use the [forums](https://discuss.huggingface.co/) to ask questions like this. Also note that there is no `finetune` script in the example folder anymore, so you should probably be using `finetune_trainer` or `run_seq2seq`.