How large is the difference?
Note that you might be running into the limited numerical precision if the max. abs difference is approx. 1e-6.
This code snippet shows that resetting the parameters as well as buffers, and calling eval() should work:
bn = nn.BatchNorm2d(3)
x = torch.randn(2, 3, 24, 24)

# change bn stats
for _ in range(10):
    out = bn(x)

# make sure output is not equal to input
print((out != x).all())
> tensor(True)

# reset
bn.reset_parameters()
bn.reset_running_stats()

print(dict(bn.named_parameters()))
> {'weight': Parameter containing:
tensor([1., 1., 1.], requires_grad=True), 'bias': Parameter containing:
tensor([0., 0., 0.], requires_grad=True)}

print(dict(bn.named_buffers()))
> {'running_mean': tensor([0., 0., 0.]), 'running_var': tensor([1., 1., 1.]), 'num_batches_tracked': tensor(0)}

bn.eval()
# test again
for _ in range(10):
    out = bn(x)

# check for equal output
print(torch.allclose(out, x)) # use allclose for numerical precision
> True