We currently don’t support elasticity of workers, which means that if one of the processes crashes with an OOM error the user is currently responsible for spawning another process and re-initializing distributed communication if they want to continue with training.
You may want to consider the use of the PyTorch elastic framework: https://github.com/pytorch/elastic 29