Solved by mrshenli in post #2 
              
              
                Yes, it is. If one process hit OOM and skipped/reran the the backward pass, it would cause de-synchronization across processes in the same group, which would lead to hang or crash. 

If so, is there a nice way to recover from this problem? 

Yep, TorchElastic is built to solve this issue. cc @Kiuk_…