One way to do this is to use a smaller timeout. The default timeout for the distributed module is 30 minutes. You can override this by specifying the timeout keyword argument to init_process_group as a timedelta type (e.g. datetime.timedelta(seconds=10)). Then if one of the processes crashes, the others will time out. The problem with your proposed solution is that you’re not guaranteed that the crashed process will come back. Therefore you’ll have to rely on some out of band mechanism to figure out which processes are still alive, and only when you know for sure you have WORLD_SIZE machines (or after adjusting WORLD_SIZE), continue and reinitialize.