Solved by ssgosh in post #7 
              
              
                From Distributed Data Parallel — PyTorch 1.8.1 documentation , under Internal Design 

Construction: The DDP constructor takes a reference to the local module, and broadcasts state_dict() from the process with rank 0 to all other processes in the group to make sure that all model replicas start from…