T_Qri:

Is it possible to hang due to oom errors during backward in several process so that the other successful processes keep waiting for them?


Yes, it is. If one process hit OOM and skipped/reran the the backward pass, it would cause de-synchronization across processes in the same group, which would lead to hang or crash.

If so, is there a nice way to recover from this problem?

Yep, TorchElastic 4 is built to solve this issue. cc @Kiuk_Chung