I was playing with momentum as you suggested. Initially, I modified the momentum directly in my code, I didn’t really notice that much of a difference in my result. Then I did this simple test.
a = torch.Tensor([[-1.5,0],[0.5,-.8]])

b = torch.nn.functional.batch_norm(a,torch.mean(a,dim=1),torch.var(a,dim=1),momentum=0.9)

I tried different values for momentum, but the output was always the same.
I also read the definition of mometum directly from https://pytorch.org/docs/stable/nn.html 10 but honestly I don’t completely understand what they mean by estimated statistic and new observed value.
Finally, I realize if I remove batch normalization at all, I get results that make sense e.g. batch size 32 is than batch size 1. Which means I am not applying batch normalization correctly. I would like if you could shed me some light regarding the usage or the exact interpretation of the momentum.
Thanks