s_n:

So does dataparallel takes this input to all devices when we use data parallel?
Because while passing input (data_var) to my model I take it to default device.
Does pytorch create copies of input image as well on all the GPUs when we use dataparallel?


nn.DataParallel will split the input tensor in dim0 and will send each chunk to the corresponding model replica on a specific device.
I.e. your model’s forward will get an input of the shape [batch_size//nb_gpus, *].
If you need to create y1 and y2 inside the forward method, you should also consider the current batch size of the input.
Also, it is correct to push the input tensor in the DataLoader loop to the default device, since nn.DataParallel will take care of the scattering and gathering of the tensors and parameters.