tree33:

The author seems to forget to add the runningmean/variance of the batchnorm layer in tensorflow to the ‘trainable variables’, so they are never really updated in the model he provided. And the is_train flag is always set to True even in evaluation.


I’m not that familiar with tensorflow, but are the running estimates treated as “trainable variables”? Since they are updated based on the current batch statistic, they won’t be updated using the gradient + optimizer.
Are you sure the running estimates are not updated, if they are not in these “trainable variables” list?



 tree33:

However, when I load the model in pytorch, then set the runningmean to all zeros and running variance to all ones, the output is different nomatter I set model to train or eval.


In your current code you are setting the weight and bias to these values: lines of code 17.
These parameters correspond to the gamma and beta parameters from the original batchnorm paper.
However, the running_mean and running_var will be initialized by default to zeroes and ones, respectively.