Solved by albanD in post #4 
              
              
                The thing is that batchnorm behaves very differently in training and eval mode. And there are many examples of models that have this exact problem. When using the saved statistics, the quality of the result is not as good.