LSTM baseline across tasks, in terms of both time to convergence and performance.
Liao & Poggio (2016) simultaneously investigated batch normalization in recurrent neural networks,
albeit only for very short sequences (10 steps). Ba et al. (2016) independently developed a variant
of batch normalization that is also applicable to recurrent neural networks and delivers similar improvements as our method.
2 P REREQUISITES
2.1 LSTM
Long Short-Term Memory (LSTM) networks are an instance of a more general class of recurrent
neural networks (RNNs), which we review brieﬂy in this paper. Given an input sequence X=
(x1;x2;:::;xT), an RNN deﬁnes a sequence of hidden states htaccording to
ht=(Whht1+Wxxt+b); (1)
where Wh2Rdhdh;Wx2Rdxdh;b2Rdhand the initial state h02Rdhare model parameters. A popular choice for the activation function ()istanh .
RNNs are popular in sequence modeling thanks to their natural ability to process variable-length
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville