Issues with finetune_trainer.py on multiple gpus
## Environment info     
- `transformers` version:3.5.1
- Platform: google cloud
- Python version: 3.7
- PyTorch version (GPU?): yes 1.6
- Tensorflow version (GPU?): - 
- Using GPU in script?: yes 
- Using distributed or parallel set-up in script?: - 

### Who can help
 Trainer: @sgugger
 Text Generation: @patrickvonplaten @TevenLeScao
 T5: @patrickvonplaten
 examples/seq2seq: @patil-suraj

## Information
Hi
I am trying to train finetune_trainer.py on multiple gpus, one issue I got is 

1) Runtime error: Input,ouptut,and indices must be on the current device, 
Looking into the codes, in training_args.py when you set device for the n_gpu > 0 on line 401, this should be changed from cuda:0 to cuda to me.

2) The accuracy on multiple gpus does not match on single gpu and this is much lower, any idea on this.

Thank you. 


Hi there. There is little anyone can do to help without knowing the actual command you are running.
<|||||>Hi
I am running
https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune_trainer.py
once
with multiple gpu machines once with 1 machine,
I have adapted it for my usecase, this is hard for me to provide the exact
command, as I need to share the whole codebase.
My question is though more general I observe performance differences if you
run this code on 1 dataset with multiple gpu/gpu, any thoughts on this?
thanks
Rabeeh

On Mon, Nov 23, 2020 at 1:16 AM Sylvain Gugger <notifications@github.com>
wrote:

> Hi there. There is little anyone can do to help without knowing the actual
> command you are running.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8718#issuecomment-731871920>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCHAF5CQRJP2LDI2XQDSRGSUPANCNFSM4T6X5WFQ>
> .
>
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.