Published as a conference paper at ICLR 2017
RECURRENT BATCH NORMALIZATION
Tim Cooijmans, Nicolas Ballas, César Laurent, Ça ˘glar Gülçehre & Aaron Courville
MILA - Université de Montréal
firstname.lastname@umontreal.ca
ABSTRACT
We propose a reparameterization of LSTM that brings the beneﬁts of batch normalization to recurrent neural networks. Whereas previous works only apply batch
normalization to the input-to-hidden transformation of RNNs, we demonstrate that
it is both possible and beneﬁcial to batch-normalize the hidden-to-hidden transition, thereby reducing internal covariate shift between time steps.
We evaluate our proposal on various sequential problems such as sequence classiﬁcation, language modeling and question answering. Our empirical results show
that our batch-normalized LSTM consistently leads to faster convergence and improved generalization.
1 I NTRODUCTION
Recurrent neural network architectures such as LSTM (Hochreiter & Schmidhuber, 1997) and
GRU (Cho et al., 2014) have recently exhibited state-of-the-art performance on a wide range of
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville