Muti GPU training with torch==1.7.0 not working
Hi,
muti GPU training with torch==1.7.0 is not working.

I have the following configuration:
- CUDA 11.0
- V100 GPU (1 or 4)
- Torch 1.7.0 (1.6.0)
- transformers 3.4.0
- Linux

When I train with one GPU all works. When I run it with 4 GPUs the CPU load goes up to 100% on one core like 1,5 MB memory is used from each GPU but the training does not start. When I downgrade torch to Version 1.6.0 everything works.

Might be an upstream problem but you might want to warn users from using torch 1.7.0

PS: Do you think I should report this at the torch repo?
I haven't noticed anything strange on my side, but didn't play much with 1.7 yet. Could you post the code you're using?<|||||>> Could you post the code you're using?

That is not possible - it is some company code. If you have some "toy" code maybe you could try it.

I am using an electra base model. Pretokenize everything. Create a torch Dataset for classification (binary).
Then I am using the normal Trainer to train...

<|||||>On AWS Sagemaker it works with Torch 1.7.0.
I think this might be an NVIDIA Driver probelm on my side...
I am closing this for now. Will reopen if needed.
... sorry ...