Mughees:

But when I wrapped the loss in DataParallel(loss) this mean and std will be created on all three devices?


No, nn.DataParallel will create model replicas on all specified devices during the forward pass.
You can verify it by printing the .device attribute of a parameter or buffer of this module in forward.



 Mughees:

And how can I make this process of moving of the data from CPU among different GPUs efficient as i often come across this problem.


Since nn.DataParallel copies the model in each iteration, we generally recommend so use DistributedDataParallel using a single process per GPU.