Since the IP was an issue, I assume you will train the model using multiple machines? If yes, the --nnode and --node_rank also need to be configured, see: https://github.com/pytorch/examples/tree/master/distributed/ddp 19


Since each process can see two devices (CUDA_VISIBLE_DEVICES=0,1), you will need to set the device_ids to make sure that each process exclusively works on one device e.g., (DistributedDataParallel(net, device_ids=[args.local_rank])).