Yes, that’s correct. The documentation covers this (the replication bit), see torch.nn.DataParallel 15. Note that this is not how the distributed version works. There, every process runs forward/backward/optimizer against a single copy of the model, so its parameters are equivalent already. Not by replicating the values, but by executing the exact same optimizer step.