Solved by Alpha in post #2 
              
              
                Hi, 
In the source code here, 
the function F.batch_norm has the parameter self.training, 
when you train model, you use model.train() 
when you test, you use model.eval() 
the model.train() tells the  self.training to be True.