Solved by Ranahanocka in post #3 
              
              
                As @SimonW replied in that thread, it’s not a bug. It’s how batch norm works. In training phase the  mean/std are learned from the training data, and loaded in the evaluation phase. 
If you turn track_running_stats off (as suggested in the post) you will instead use the mean and std of the batch in …