Hi, I have 2 GPUs running a standard DCGAN model and I have looked at some of the results and they seem a little different from the times I have run the model on a single GPU. I have just a few questions with respect to the GPU parallelism.


What to do with the loss? Can we just throw the loss on a device like such: MSELoss().cuda() and it will work? Why would we want or don’t want to wrap the loss in DataParallel()?


If we wrap the loss in DataParallel() with 2 GPUs we get a loss scalar from each device. Can we just average them to get a scalar to perform backpropagation or there is a better way to handle that?


Tutorial here https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html 4 shows the following usage of .to():


model = Model(input_size, output_size)
if torch.cuda.device_count() > 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs
  model = nn.DataParallel(model)

model.to(device)

However, the documentation (https://pytorch.org/docs/stable/tensors.html#torch.Tensor.to) says: Returns a Tensor with the specified device and (optional) dtype. So does .to() modify the the model or tensor in place, or we have to reassign it back like so: model = model.to(device)?
I have packed 3 questions in 1 post since they all seem highly related to me. Thank you for attention.
UPDATE: Question 3 is answered by looking in the right place over here: https://pytorch.org/docs/stable/nn.html#torch.nn.Module.to 4. This method modifies the module in-place.