alldbi:

According to the expected behavior of batchnorm, its output should be the same in eval and training modes if the running stats are equal.


No, that’s not true.
During training, the input activation will be normalized using the batch statistics (so the stats calculated from the input activation itself) and the running stats will be updated using the momentum, the current batch stats, and the old running stats.
During evaluation, the input activation will be normalized using the running stats.