But .to(device) is called two times where device is always the gpu+rank. So the model is sent with two different initializations at first to both gpus, but what I am wondering now is if DistributedDataParallel(model) syncs the initialization from rank 0 to rank 1?