It seems that the program got stuck before the eval phase, as you didn’t see anything printed out from the eval phase.
I guess the issue is this line torch.cuda.set_device(gpu_ids[0]), since gpu_ids[0] is always 0, then you set device rank as 0 even for non-master processes, and hence the collective communication incurred by DDP will hang.
I think you can try: 1) remove torch.cuda.set_device(gpu_ids[0]) from main method, and 2) add torch.cuda.set_device(rank) at the beginning of train method.