I am trying to design a network which contains a recurrent sub-network. The input of this sub-network will change over iterations(because it current input depends on its last output). However, I find the network works much worse in the evaluation mode than the training mode. Through debugging, I find it’s the problem of batchnorm statistics, i.e. running_mean and running_var. As stated in recent paper “recurrent batch normalization” (https://arxiv.org/pdf/1603.09025.pdf 9), the batchnorm statistics should be done separately for each iterations. I have also saw one implementation of this paper https://github.com/jihunchoi/recurrent-batch-normalization-pytorch/blob/master/bnlstm.py 10, but I don’t think it works since it doesn’t implement the important separate batch stastics for different iterations. Does someone know how to implement this or other tricks that have the equivalent effect?