Tmax, we use the population statistic of time Tmax for all time steps beyond it.
During training we estimate the statistics across the minibatch, independently for each timestep. At
test time we use estimates obtained by averaging the minibatch estimates over the training set.
4 I NITIALIZING 
FOR GRADIENT FLOW
Although batch normalization allows for easy control of the pre-activation variance through the 
parameters, common practice is to normalize to unit variance. We suspect that the previous difﬁculties with recurrent batch normalization reported in Laurent et al. (2016); Amodei et al. (2015) are
largely due to improper initialization of the batch normalization parameters, and 
in particular. In
this section we demonstrate the impact of 
on gradient ﬂow.
0 100 200 300 400 500 600 700 800
t10-2610-2410-2210-2010-1810-1610-1410-1210-1010-810-610-410-2100||dloss/dh_t||_2RNN gradient propagation
gamma=0.10
gamma=0.20
gamma=0.30
gamma=0.40
gamma=0.50
gamma=0.60
gamma=0.70
gamma=0.80
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville