Hello!
I am using DistributedDataParallel on one node with 4 GPUs. Also using DistributedSampler for training.
self.model = torch.nn.parallel.DistributedDataParallel(
    self.model,
    device_ids=[self.local_rank],
    output_device=self.local_rank,
    find_unused_parameters=True
)

Doing evaluation after every train epoch only on rank_0.
During evaluation I observed (through nvidia-smi) that other (1, 2, 3) ranks/gpus continue to be processing something with 100% load.
My questions:

Is it possible that other ranks continuing training next epoch without waiting for rank_0 to finish evaluation?
In case (1) is true, is it ok to leave it like this (will rank_0 process it’s part of the next epoch after it finishes evaluation)? Or is it better to set a barrier so that other ranks will wait for rank_0 to do evaluation?

Thanks!