Per my understanding, the model is already initialized upon your call to Model(). Then .to(device) will move it to the specified device (which will probably be the first gpu on the first machine).
The call DistributedDataParallel(model) will replicate your model’s params on the machines and devices you’ve specified.
So it starts from the same initiaization. Moreover, the computed gradients for your model’s params on each machine/device will be sent back to the first machine/device and summed.