From Distributed Data Parallel — PyTorch 1.8.1 documentation 1 , under Internal Design

Construction: The DDP constructor takes a reference to the local module, and broadcasts state_dict() from the process with rank 0 to all other processes in the group to make sure that all model replicas start from the exact same state.

Hope this helps.