Your program seems to be correct, some questions:

what backend are you using?

and could you please add these tests?:

change device to “cpu”, will the error be the same?
print rank, train_loss, train_acc, val_loss, val_acc in each of your process, before all_reduce


theoretically this problem should not happen

the default gloo backend supports gpu all_reduce and broadcast

you cannot repeat or let out a process, since all_reduce is blocking