C. Laurent, G. Pereyra, P. Brakel, Y . Zhang, and Y . Bengio. Batch normalized recurrent neural
networks. ICASSP , 2016.
Quoc V Le, N. Jaitly, and G. Hinton. A simple way to initialize recurrent networks of rectiﬁed linear
units. arXiv:1504.00941 , 2015.
Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural
networks and visual cortex. arXiv:1604.03640 , 2016.
M. Mahoney. Large text compression benchmark. 2009.
M. P. Marcus, M. Marcinkiewicz, and B. Santorini. Building a large annotated corpus of english:
The penn treebank. Comput. Linguist. , 1993.
J. Martens and I. Sutskever. Learning recurrent neural networks with hessian-free optimization. In
ICML , 2011.
T. Mikolov, I. Sutskever, A. Deoras, H. Le, S. Kombrink, and J. Cernocky. Subword language
modeling with neural networks. preprint , 2012.
Yann Ollivier. Persistent contextual neural networks for learning symbolic data sequences. CoRR ,
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville