Solved by mrshenli in post #2 
              
              
                Hey @Scott_Hoang, yes, this is the expected behavior, as OOM in one of the process will lead to AllReduce communication desync across the entire DDP gang.  https://pytorch.org/elastic is  built to solve this problem. It will destruct all DDP instances across all processes, reconstruct a new gang, an…