The batch normalization transform relies on batch statistics to standardize the LSTM activations. It
would seem natural to share the statistics that are used for normalization across time, just as recurrent
neural networks share their parameters over time. However, we ﬁnd that simply averaging statistics
over time severely degrades performance. Although LSTM activations do converge to a stationary
distribution, we observe that their statistics during the initial transient differ signiﬁcantly (see Figure 5 in Appendix A). Consequently, we recommend using separate statistics for each timestep to
preserve information of the initial transient phase in the activations.1
Generalizing the model to sequences longer than those seen during training is straightforward thanks
to the rapid convergence of the activations to their steady-state distributions (cf. Figure 5). For our
experiments we estimate the population statistics separately for each timestep 1;:::;Tmax where
1Note that we separate only the statistics over time and not the 
andparameters.
3
Published as a conference paper at ICLR 2017
Tmax is the length of the longest training sequence. When at test time we need to generalize beyond
Tmax, we use the population statistic of time Tmax for all time steps beyond it.
During training we estimate the statistics across the minibatch, independently for each timestep. At
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville