Does your model forward pass contains any communication? (e.g., SyncBatchNorm layers)
BTW, how long does the validation take? Would I be correct if I assume the log shows that rank 0 finishes barrier successfully but rank 1 doesn’t?