How did you create the distributed training procedure?
Is it a manual approach or did you use nn.DistributedDataParallel?
In the latter case, synchronization points are automatically in the training process.
From the docs 2:

Across processes, DDP inserts necessary parameter synchronizations in forward passes and gradient synchronizations in backward passes.