Last experiment:
(1) test set DIFFERENT from  train set but originating both from a large data set
(2) with data augmentation + shuffling for Train set,
(3) still no shuffling and no data augmentation for test set
(4) BN with track_running_stats=False

So if I am correct the fact that the test loss is following the train loss is due to  track_running_stats=False for the Batch Norm layers at least all ones. May be some BN layers are more concerned than others? Anyway is it correct to have this switch turned to False ???