It depends how you specify the momentum and note that PyTorch uses a probably unexpected default value of 0.1.
I’ve implemented the manual batchnorm approach to stick to the PyTorch definition. From the docs:

This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is x_hat_new = (1 - momentum) * x_hat + momentum x_t , where x_hat is the estimated statistic and x_t is the new observed value.