Hi,
I have a short question about the behavior of the initial model initialization with DDP.
Let’s say I use two GPUs, and my code looks something like this:
model = Model().to(device)
model = DistributedDataParallel(model)

My question now is whether the model is initialized two times separately, or whether the model as initialized on rank 0 is mirrored on rank 1.
So are both GPUs starting from the same initialization or from a different one assuming the seeds are different?