Research Chairs and CIFAR. Experiments were carried out using the Theano (Team et al., 2016) and
the Blocks and Fuel (van Merriënboer et al., 2015) libraries for scientiﬁc computing. We thank David
Krueger, Saizheng Zhang, Ishmael Belghazi and Yoshua Bengio for discussions and suggestions.
REFERENCES
D. Amodei et al. Deep speech 2: End-to-end speech recognition in english and mandarin.
arXiv:1512.02595 , 2015.
M. Arjovsky, A. Shah, and Y . Bengio. Unitary evolution recurrent neural networks.
arXiv:1511.06464 , 2015.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06450 ,
2016.
D. Bahdanau, K. Cho, and Y . Bengio. Neural machine translation by jointly learning to align and
translate. ICLR , 2015.
Y . Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is
difﬁcult. Neural Networks, IEEE Transactions on , 1994.
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville