Ideally your performance should scale with the number of GPUs.
What kind of performance do you measure using multiple GPUs vs. single one?



 isalirezag:

not sure what does it mean to reduce the loss


You could calculate the mean or sum of the loss and call backward() on the result.