Adding a bit more context to @mrshenli’s comments, you could try to reset the DDP state by calling destroy_process_group() and re-initializing it, however that doesn’t guarantee that your tensors (distributed among multiple workers) are also reset. In short, a complete state reset on the worker is application dependent (and often non-trivial). For transient exceptions you can use torchelastic to launch your workers, and just let the worker process throw the exception out and fail. Elastic will monitor the worker pids and will restart the world if it detects that one (or more) workers have failed. Note, that due to this behavior, (assuming you have checkpoints) you will lose progress between checkpoints.