Hello.
I’d like to learn the motivation behind batchnorm momentum set to 0.1 by default. Seems like keeping running average with such a low weight goes against common sense. Also other libraries like tf 20 or lasagne 5 have this value by default at 0.99 or 0.9 respectably. By momentum I mean the following value:
X_{mov_avg} = X_{mov_avg} * momentum + X_{mean} * (1 - momentum)
Is there some work I’m not familiar with that suggests better results with low momentum inside dataset statistics computation? Or maybe there’s a typo in description of the module and you meant alpha? Meaning that momentum = 1 - alpha. Otherwise maybe it makes sense to change the default value to a more reasonable one?
Thanks for the answer.