Solved by ptrblck in post #7 
              
              
                It depends how you specify the momentum and note that PyTorch uses a probably unexpected default value of 0.1. 
I’ve implemented the manual batchnorm approach to stick to the PyTorch definition. From the docs: 

This momentum argument is different from one used in optimizer classes and the conventio…