Hey @EricWiener,
TorchElastic is designed to recover from such errors, see: Torch Distributed Elastic — PyTorch 1.9.0 documentation 1
If you just need the non-failing processes to crash when any peer hit OOM, you can set a timeout when calling init_process_group, and set NCCL_ASYNC_ERROR_HANDLING env var when using NCCL backend. see the timeout argument docstring for init_process_group at Distributed communication package - torch.distributed — PyTorch master documentation 2