There seems always one GPU got stuck whose utilization is 0%, and the others are waiting for it to synchronizing.

Do you have uneven inputs? For example, you run one epoch every few seconds, and one partition on a rank has much fewer batches to process.

This issue disappears after switching to another server (with the same image).

Does the “image” here means cuda:6 still has 0 GPU utilization for some time, but you don’t feel it’s get stuck?