Hi,
According to the expected behavior of batchnorm, its output should be the same in eval and training modes if the running stats are equal. However, I do not get consistent outputs when the stats are the same. For instance, please consider the following toy example in which the outputs of two exact batchnorm modules are not the same although running stats are exactly the same.
import copy
import torch
import torch.nn as nn

bn1 = nn.BatchNorm2d(3)
bn2 = copy.deepcopy(bn1)

# create dummy input
img = torch.rand(size=[4, 3, 32, 32])

# set the first bn in training mode
bn1.train()
# compute a dummy output
pred1 = bn1(img).sum()

print('bn1 running mean:', bn1.running_mean)
print('bn1 running var:', bn1.running_var)
print('output1:', pred1.item(), '\n')

# set the second bn in training mode
bn2.train()
# update batchnorm stats
_ = bn2(img)

print('bn2 running mean:', bn2.running_mean)
print('bn2 running var:', bn2.running_var)

# set the bn to eval mode
bn2.eval()
pred2 = bn2(img).sum()
print('output2:', pred2.item())

Example output is:
bn1 running mean: tensor([0.0499, 0.0501, 0.0499])
bn1 running var: tensor([0.9081, 0.9082, 0.9083])
output1: -0.00052642822265625 

bn2 running mean: tensor([0.0499, 0.0501, 0.0499])
bn2 running var: tensor([0.9081, 0.9082, 0.9083])
output2: 5794.7978515625

It is expected to get equal outputs but they are drastically different. Can anyone help me understand what is causing the difference in the outputs?