This is a very strange behaviour. I think this is a bug.
If in training batch_mean and batch_std are used, and in eval, running_mean and runnning_std are used, then how is it possible to track the convergence?
I am trying to train a classifier. In the last batches of the first epoch I start to see good results. (batches which the network did not see before.)
However, when moving to evaluation mode - the results are terrible.
I think that the proper way of addressing train(true) for batch normalization is to apply the updated running_mean and running_std, and not the current batch mean and std.