All communication done through torch.distributed is collective, meaning all processes expect all their peers to participate in all collective calls they execute. If a single processes ends up not participating, the others will time out or raise an exception. The only way out of this is to let all processes timeout or fail and to reinitialize the distributed module.
You can use torch.distributed.destroy_process_group to deinitialize and then make another call to torch.distributed.init_process_group to reinitialize. This can only work if you’re using either the Gloo or the NCCL backend, and that the underlying initialization method can be reused. I believe this is the case for the the file initialization method as well as the TCP initialization method (see https://pytorch.org/docs/stable/distributed.html#initialization 33 for more information on both).
Good luck!