Hi, everyone
When I train my model with DDP, I observe that my training process got stuck every few seconds. The device information is shown in the following figure when it is stuck.

image637×643 26.2 KB

There seems always one GPU got stuck whose utilization is 0%, and the others are waiting for it to synchronizing.
This issue disappears after switching to another server (with the same image).
Though it is solved, I am curious about the reasons. I will appreciate it if someone shares the ideas.