Solved by ptrblck in post #2 
              
              
                track_running_stats is used to initialize the running estimates as well as to check if they should be updated in training (line of code). 
The running estimates won’t be updated in eval: 
bn = nn.BatchNorm2d(3)
for _ in range(10):
    x = torch.randn(10, 3, 24, 24)
    out = bn(x)
print(bn.running_m…