Vibhatha_Abeykoon:

About the second point, so this would be the case to support the functionality
of some specific layers like running_mean in BatchNorm layers. But for
general cases, this is also skipped, so there won’t be such synchs.
Is this an assumption that we can make when we use DDP?


Yep, this is correct.