Hey @qianyizhang
Yes, the error is expected. Because, say you have two DDP processes, X and Y. If process X hits a OOM error in the forward pass of one iteration but Y runs correctly, as a result, X would skip its backward pass in that iteration causing a de-synchronization. DDP itself cannot recover from this error.
However, torchelastic 3 is built to solve this exact problem. It would kill the entire DDP gang, reconstruct a new DDP gang, and revert to the previous checkpoint when such OOM occurs. cc @Kiuk_Chung