When trying to train a model in DDP which has a sync_batch_norm layer with uneven inputs, the suggestion is to use the parameter throw_on_early_termination as True within the join() context manager of DDP. I wanted to know what is the best way to handle the exception raised by the  join() context manager in case one rank exhausts all its input and we just want the training to continue normally for next iterations? Should we just swallow the exception and move forward? Should we put a CUDA synchronize call as part of the exception handling? Anything else?