@Kiuk_Chung I see.
But is it possible to reset DDP state on one worker (with NCCL backend)? Without resetting everything, which is somewhat expansive.
After all, in my case the connection is not lost, it’s simply halted with OOM. The ideal scenario could be:
(assuming a bad case where bucket 1 gradients already reduced, and stuck on bucket2 with worker#0 OOM)

worker#0 rerun another batch and only sync bucket2 + later, to “catch up” with the rest.
worker#0 continue to sync while only sending 0.0 gradients without further computation.

I imagine solution#1 is very tricky if not impossible, whereas solution#2 is very much feasible?
If so, can you point me some directions to make it work? (files, watch-outs, etc.) Thanks!