0
BB@~ft
~it
~ot
~gt1
CCA=Whht1+Wxxt+b (2)
ct=(~ft)ct1+(~it)tanh( ~gt) (3)
ht=(~ot)tanh(ct); (4)
where Wh2Rdh4dh;WxRdx4dh;b2R4dhand the initial states h02Rdh;c02Rdhare
model parameters. is the logistic sigmoid function, and the operator denotes the Hadamard
product.
The LSTM differs from simple RNNs in that it has an additional memory cellctwhose update is
nearly linear which allows the gradient to ﬂow back through time more easily. In addition, unlike
the RNN which overwrites its content at each timestep, the update of the LSTM cell is regulated by
a set of gates. The forget gate ftdetermines the extent to which information is carried over from the
previous timestep, and the input gate itcontrols the ﬂow of information from the current input xt.
The output gate otallows the model to read from the cell. This carefully controlled interaction with
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville