How did you narrow down the batchnorm layers? Did you replace them and got a zero absolute error or what’s the reasoning?
Due to the limited floating point precision operations such as e.g. sum will also suffer from this effect:
x = torch.randn(100, 100, 100)
sum1 = x.sum()
sum2 = x.sum(0).sum(0).sum(0)
print((sum1 - sum2).abs())
> tensor(0.0016)