Euruson:

Any references for more details? The tutorial on the official website just mentions that the backward would trigger the barrier.


I don’t think this is documented, but under certain conditions there might be a sync during the forward pass: https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L675 16. In addition to this, we rebuild buckets once: https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L671 5, which triggers a sync across ranks: https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/reducer.cpp#L1377 4