Distributed device ordinal question
## ❓ Questions & Help

In the following line, we set the device ordinal to be local rank. However, suppose we have four independent nodes each with only one GPU. Then the 4th node (rank 3) will execute this line with the device ordinal 3, but it really only has 1 GPU, so it'll be invalid to ask for the GPU 3. So won't that break things? Shouldn't it be `torch.device("cuda", 0)`?

https://github.com/huggingface/pytorch-transformers/blob/0287d264e913e10018a95a2723115dc9121e5fc6/examples/run_glue.py#L403
No, `local_rank` is only the local rank on each node.<|||||>Ah I see, thanks! A tangential question, considering the distributed setting only, would it be the same if we simply call `.cuda()` for the model and tensors instead of passing around the device?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
