This should be fine, because DistributedDataParallel broadcasts model states from rank 0 to all other ranks at construction time. See the code below:

  
      github.com
  
  
    pytorch/pytorch/blob/34284c127930dc12d612c47cab44cf09b432b522/torch/nn/parallel/distributed.py#L280-L285 18

# Sync params and buffers
module_states = list(self.module.state_dict().values())
if len(module_states) > 0:
    self._distributed_broadcast_coalesced(
        module_states,
        self.broadcast_bucket_size)