Bug in multi-gpu training setting max_iters 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.3
- Platform:
- Python version: 3.8
- PyTorch version (GPU?): 1.8
- Tensorflow version (GPU?): - 
- Using GPU in script?: yes 
- Using distributed or parallel set-up in script?: yes 

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger,  @patrickvonplaten, @patil-suraj

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

 trainer: @sgugger

## Information

I am training T5 model using the command in the repo on 4 GPUs in a distributed way, the issue arises that if one set max_iters then the number of iterations with 4 GPUs is not divided by 4 anymore, only one get speed up if max_iters is not set, and this looks like this is a bug. 


## To reproduce

Steps to reproduce the behavior:
Please run 
python -m torch.distributed.launch \
    --nproc_per_node 8 \
    examples/seq2seq/run_summarization.py \
    --model_name_or_path t5-small \
    --do_train \
    --do_eval \
    --dataset_name xsum \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --overwrite_output_dir \
    --predict_with_generate \
    --max_train_samples 500 \
    --max_val_samples 500
   --max_iters 100 

compare the results with the case you run on 1 GPU, both would have the same number of iterations to get completed once running which is not correct 


## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

There is no `--max_iters` argument in the `run_summarization` script, so I'm not sure what you're referring.<|||||>Hi
I apologize for the typo, this is max_steps, if you set it and run a code
in a distributed way and compare it with non-distributed way, the number of
steps would not differ, but if you try with setting max_train_epochs, you
would see less number of iterations when training on multiple GPUs, meaning that the code is
correctly setting the parameters in that case. thanks

On Wed, Mar 17, 2021 at 2:08 PM Sylvain Gugger ***@***.***>
wrote:

> There is no --max_iters argument in the run_summarization script, so I'm
> not sure what you're referring.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/10768#issuecomment-801066552>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS45N4YLS7NRPPJ32GUWPW3TECSTBANCNFSM4ZKGUFPA>
> .
>
<|||||>Yes, `max_steps` is the number of training steps, so whether you run on one or several GPUs, you will do that number of training steps. That is the intended behavior and it is not a bug.
 
`num_epochs` is the number of training epochs. Depending on your number of GPUs you will not have the same number of training steps per epoch (as long as you keep `per_device_train_batch_size` the same) so you will not train for the same number of total steps.<|||||>Hi
thanks for the response, still to me if a user needs max_steps on multiple
gpus, it needs to become a smaller number as this divides per number of
gpus, similar to number of epochs.

On Thu, Mar 18, 2021 at 3:36 PM Sylvain Gugger ***@***.***>
wrote:

> Yes, max_steps is the number of training steps, so whether you run on one
> or several GPUs, you will do that number of training steps. That is the
> intended behavior and it is not a bug.
>
> num_epochs is the number of training epochs. Depending on your number of
> GPUs you will not have the same number of training steps per epoch (as long
> as you keep per_device_train_batch_size the same) so you will not train
> for the same number of total steps.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/10768#issuecomment-801980651>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS45N4ZF2CRND434WMFIIUDTEIFYBANCNFSM4ZKGUFPA>
> .
>
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.