Yes. At the time of DDP wrapping, parameters and buffers (i.e., model.state_dict()) on rank0 are broadcasted to all other ranks. So although weights on different ranks were initialized from different seeds, they will have the same starting point (rank0’s state) after DDP construction.