Ideally your performance should scale with the number of GPUs.


it was a classification task, the performance got worse with increasing gpus which was not making sense to me.


You could calculate the mean or sum of the loss and call backward() on the result.


I did do that, in the code that i posted I used  loss.sum().backward(), isnot that what you mean?