I think you shouldn’t use  CUDA_VISIBLE_DEVICES=0,1,2  as not all 3 GPUs are bot connected and visible to the same system.

Yes, you’re right.
I’m following the training similar to the one mentioned here : https://github.com/pytorch/vision/tree/v0.4.0/references/detection 3
Is there a way I can train across multiple nodes in pytorch?