Solved by albanD in post #2 
              
              
                Hi, 
When using batchnorm in training mode, the running stats are always updated yes. 
You should be using the eval mode to use these stats and stop updating them when evaluating indeed.