@mrshenli. Thank you for the response.
I see. The first point is clear to me.
About the second point, so this would be the case to support the functionality
of some specific layers like running_mean in BatchNorm layers.  But for
general cases, this is also skipped, so there won’t be such synchs.
Is this an assumption that we can make when we use DDP?