I see only one log line for Test Epoch 0, Top1 Accuracy:0.08558, Top5 Accuracy:0.22848, Loss:5.3841 , is it possible that rank 0 is issuing some additional collective comm. as part of this (e.g. all_reduce) that isn’t matched by the other ranks? That may explain the hang.