The running stats are updated during the forward pass as seen here:
bn = nn.BatchNorm2d(3)
x = torch.randn(16, 3, 24, 24)
print(bn.running_mean)
> tensor([0., 0., 0.])

print(bn.running_var)
> tensor([1., 1., 1.])

out = bn(x)
print(bn.running_mean)
> tensor([ 0.0002, -0.0007,  0.0006])

print(bn.running_var)
> tensor([0.9991, 1.0022, 0.9988])