Why is  dist.barrier()  not necessary in the above reference example?

IIUC, the torchvision example only saves the checkpoint to file in epoch but is not reading from it unless it is recovering from a crash? In that case it is not a hard requirement to perform the barrier, because

it does not need to ensure non-master processes are reading stale checkpoints.
non-master processes will just block on DDP backward (AllReduce) and waiting for the master to join.


Doesn’t this mean that the  evaluate()  function is called on  all  the processes (i.e. all the GPUs in this case)?

cc @fmassa for this implementation

But then, again, shouldn’t we wait, using  dist.barrier() , for all the processes to finish the computations and for the master to gather the gradients, before evaluating the model?

The gradients are synchronized in DDP backward using AllReduce operations. So, there is no need to add another barrier here to do that. As soon as loss.backward() returns, the local gradients should be representing the global average. However, it might need a barrier here for a different reason. If the evaluate() step takes too long, non-master processes could timeout on AllReduce. If that happens, barrier might help.