modeling with neural networks. preprint , 2012.
Yann Ollivier. Persistent contextual neural networks for learning symbolic data sequences. CoRR ,
abs/1306.0514, 2013.
Marius Pachitariu and Maneesh Sahani. Regularization and nonlinearities for neural language models: when are they needed? arXiv:1301.5650 , 2013.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difﬁculty of training recurrent neural
networks. arXiv:1211.5063 , 2012.
H. Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood
function. Journal of statistical planning and inference , 2000.
The Theano Development Team et al. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints , abs/1605.02688, May 2016.
T. Tieleman and G. Hinton. Lecture 6.5—RmsProp: Divide the gradient by a running average of its
recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville