Ah, I did not realize ToTensor() would also do normalization. I managed to bridge the performance difference, although it is slightly annoying that torch.mean(data, [0, 2, 3]) is possible but torch.std(data, [0, 2, 3]) is not.