If the collective operation fails (with an OOM or other exception) and you are able to catch it you may be able to reissue the operation. Whether this works or not depends on whether the distributed backend is still in a good state or not. With NCCL once the state goes bad you have to destroy the process group and re initialize it. You could try to salvage your processes but you’d have to call destroy and initialize on all your workers together. There may be other states in your application that goes out of sync whenever there is an exception only observed in a subset of your workers. if you are able to recover that state you can try to restore the workers into a well defined point in time. Based on what we’ve observed with distributed applications it’s non trivial to restore the full application state (ddp + user code) in a distributed setting and the cleanest restore is to tear the processes down and restore from a checkpoint. This is why elastic was designed as such. Trade off is restart overhead versus correctness and maintainability. Note that with elastic you are simply restarting your worker processes so the penalty you pay is your initialization time (loading the model, allocating mem for data, etc) and not actually restarting the node or container.