amsword:

One more question is that if the network has some issues on those machines or straggler issues, would it be possible that the GPU is still utilizing 98%~100%? The GPUs are fully utilized and i was thinking there is no network issue.


Not 100% sure, but if GPU reports block waiting for AllReduce as busy, then slow network or straggler could lead to 100% utilization for the entire group. This can be measured by submitting a job that only runs allreduce. BTW, based one past observations, GPU would (sometimes?) report 100% utilization even if DDP hangs. So I think this is possible.
cc @ngimel in case you know how CUDA would behave here