Hey @Scott_Hoang, yes, this is the expected behavior, as OOM in one of the process will lead to AllReduce communication desync across the entire DDP gang.  https://pytorch.org/elastic 24 is  built to solve this problem. It will destruct all DDP instances across all processes, reconstruct a new gang, and then recover from the previous checkpoint.
cc @Kiuk_Chung