Thanks for the help! I got the idea, but how can I “let all processes timeout or fail”. How can all the processes know that there is one process meeting an exception and all the processes should destroy_process_group and reinitialize? If one process meet an exception for one minibatch, can all the processes simply just jump the current minibatch and run the next minibatch?
I find the following snippets in pytorch repo which might be helpful, but not sure how to implement the idea in detail.
def test_barrier_timeout_global(self):
        dist.destroy_process_group()

        # Explicitly pass world size to the barrier because we've
        # just destroyed any state in torch.distributed.
        self._barrier(wait_for=int(WORLD_SIZE))

        # Reinitialize global process group
        timeout = timedelta(seconds=0.2)
        dist.init_process_group(
            init_method=INIT_METHOD,
            backend=BACKEND,
            world_size=int(WORLD_SIZE),
            rank=self.rank,
            timeout=timeout,
        )
self._test_barrier_timeout(dist.group.WORLD, timeout)

(from test_distributed.py 14)