Hi @Yozey
Based on the pointer you provided, in TensorFlow pop_mean and pop_var are updated adaptive during the model training step (batch by batch) based on batch_mean and batch_var for current batch with some decay (say 0.99). And during the test step, pop_mean/var will be used directly for model evaluation.
Based on my understanding, PyTorch dose exactly the same thing, which we can know by reading this part of comment (https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py#L107-L110 93). If I’m correct, the momentum is used as the decay factor in PyTorch. If you delve a bit into the source code, you can find here (https://github.com/pytorch/pytorch/blob/2502ac082b74bdcbe95826ecb56d1416b344ef0d/torch/csrc/autograd/functions/batch_normalization.cpp#L66-L93 48), running_mean/var calculated for a particular batch are first saved in some place i.e. saved_mean/var and used in further forward function. Thus, it’s probably the case that in the forward function, saved_mean/var will be aggregated into running_mean/var with decay.
Therefore, everything till now is the same in TensorFlow. My thinking is you don’t need to update running_mean/var manually, PyTorch will do that for you.
Hope this one helps
Thanks