timesteps. RNNs are deeper in the time direction, and as such batch normalization would be most
beneﬁcial when applied horizontally. However, Laurent et al. (2016) hypothesized that applying
batch normalization in this way hurts training because of exploding gradients due to repeated rescaling.
Our ﬁndings run counter to this hypothesis. We show that it is both possible and highly beneﬁcial to
apply batch normalization in the hidden-to-hidden transition of recurrent models. In particular, we
describe a reparameterization of LSTM (Section 3) that involves batch normalization and demonstrate that it is easier to optimize and generalizes better. In addition, we empirically analyze the
1arXiv:1603.09025v5  [cs.LG]  28 Feb 2017
Published as a conference paper at ICLR 2017
gradient backpropagation and show that proper initialization of the batch normalization parameters
is crucial to avoiding vanishing gradient (Section 4). We evaluate our proposal on several sequential problems and show (Section 5) that our LSTM reparameterization consistently outperforms the
LSTM baseline across tasks, in terms of both time to convergence and performance.
Liao & Poggio (2016) simultaneously investigated batch normalization in recurrent neural networks,
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville