O interesting! So if loss is defined inside the model, then loss can have independent values across different GPUs, and in that case we would need to add them up (or use mean) to convert them to a scalar value, and then call .backward().