It does not matter how many GPUs you use. If loss is already a scalar, then you can just call backward loss.backward() but if it is not scalar, then you can convert that to a scalar and then call backward: loss.mean().backward(). The number of GPUs do not play any role on how to call the backward function.