the early-stopped models.
8
Published as a conference paper at ICLR 2017
6 C ONCLUSION
Contrary to previous ﬁndings by Laurent et al. (2016); Amodei et al. (2015), we have demonstrated
that batch-normalizing the hidden states of recurrent neural networks greatly improves optimization. Indeed, doing so yields beneﬁts similar to those of batch normalization in feed-forward neural
networks: our proposed BN-LSTM trains faster and generalizes better on a variety of tasks including language modeling and question-answering. We have argued that proper initialization of
the batch normalization parameters is crucial, and suggest that previous difﬁculties (Laurent et al.,
2016; Amodei et al., 2015) were due in large part to improper initialization. Finally, we have shown
our model to apply to complex settings involving variable-length data, bidirectionality and highly
nonlinear attention mechanisms.
ACKNOWLEDGEMENTS
The authors would like to acknowledge the following agencies for research funding and computing
support: the Nuance Foundation, Samsung, NSERC, Calcul Québec, Compute Canada, the Canada
Research Chairs and CIFAR. Experiments were carried out using the Theano (Team et al., 2016) and
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville