It would be a good idea to review the blog post suggested by ptrblck above.
In DataParallel, parallel_apply does not perform parameter update or synchronization.
replicate function synchronizes model parameters across GPUs.
dataparallel forward 1
As I noted, weights on GPU 0 is saved.
You can almost consider weights on other GPUs to be not really there constantly. Parameters on other GPUS are only created just before their forward computation by the above replicate function. And the replicate function is called at every forward call.