Hi all,
I wonder how to reduce the params of the model when I use torch.nn.parallel.DistributedDataParallel. Looking the code here 5, there is no explicit all_reduce in the training loop. According to the docs for this module 1 there is a note for this as following:

Parameters are never broadcast between processes. The module performs an all-reduce step on gradients and assumes that they will be modified by the optimizer in all processes in the same way. Buffers (e.g. BatchNorm stats) are broadcast from the module in process of rank 0, to all other replicas in the system in every iteration.

But it’s unclear that the module performs all-reduce automatically or not. What’s the difference of broadcasting the parameters and broadcasting the gradients?