Even slower when using multiple gpus with sharded_ddp
## Environment info

- `transformers` version: 4.2.2
- Platform: Linux-3.10.0-1062.el7.x86_64-x86_64-with-centos-7.7.1908-Core
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: True

### Who can help

Library:
- trainer: @sgugger

 -->

## Information

Model I am using (Bert, XLNet ...): facebook/bart-large-cnn

## To reproduce

Steps to reproduce the behavior:

When I was trying to use sharded_ddp and multiple GPUs to accelerate the training process, my command I used is as follows:

CUDA_VISIBLE_DEIVCES=6,7 python -m torch.distributed.launch --nproc_per_node=2 finetune_trainer.py \
--sharded_ddp --[other_args]

However, the experiment even took a longer time than the experiment using a single GPU. May I ask which part I did was wrong?


`--sharded_ddp` is not there to accelerate your training, it's there to save GPU memory (for very large models) at some cost on the training time. So if you can finetune on one GPU, you should definitely use this option.<|||||>@sgugger  as long as I know sharded_ddp is only for distributed training, not sure why you have suggested to use sharded_ddp on one GPU? <|||||>I have not suggested that. I have said to just fine-tune your model on one GPU without any kind of DDP (so no `--sharded_ddp`). It does not make any sense to use this option if your model and its training can fit on one GPU as it is there to reduce GPU memory, not speed up training.<|||||>thanks a lot, now I understood what you meant.

On Mon, Mar 22, 2021 at 3:08 AM Sylvain Gugger ***@***.***>
wrote:

> I have not suggested that. I have said to just fine-tune your model on one
> GPU without any kind of DDP (so no --sharded_ddp). It does not make any
> sense to use this option if your model and its training can fit on one GPU
> as it is there to reduce GPU memory, not speed up training.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/10758#issuecomment-803714561>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMRT24IR4P3H5QAT753TE2RCVANCNFSM4ZJW56GA>
> .
>
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.