Solved by Eta_C in post #2 
              
              
                running_mean = (1 - momentum) * running_mean + momentum * x_mean

https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html?highlight=batchnorm#torch.nn.BatchNorm2d 
.. note::
        This :attr:`momentum` argument is different from one used in optimizer
        classes and the conventiona…