Dear @ptrblck,
Below  are a series of experiments with resnet20, batch_size=128 both for training and testing.
First, let consider: Same data for train and test, no data augmentation (ie. no random flip H/V, rotations 90,180,270), and BN track_running_stats=False.

Here the two losses are pretty the same after 3 epochs. Ok, now I will turn the train shuffling ON