I have tried to reproduce in an even simpler setting, but apparently this happens only when using two GPUs. Doing all the operations on a single GPU gets the same timing for both F.mse_loss and writing it explicitly in terms of difference, power and mean.