The output gate otallows the model to read from the cell. This carefully controlled interaction with
the cell is what allows the LSTM to robustly retain information for long periods of time.
2.2 B ATCH NORMALIZATION
Covariate shift (Shimodaira, 2000) is a phenomenon in machine learning where the features presented to a model change in distribution. In order for learning to succeed in the presence of covariate shift, the model’s parameters must be adjusted not just to learn the concept at hand but also to
adapt to the changing distribution of the inputs. In deep neural networks, this problem manifests as
2
Published as a conference paper at ICLR 2017
internal covariate shift (Ioffe & Szegedy, 2015), where changing the parameters of a layer affects
the distribution of the inputs to all layers above it.
Batch Normalization (Ioffe & Szegedy, 2015) is a recently proposed network reparameterization
which aims to reduce internal covariate shift. It does so by standardizing the activations using
empirical estimates of their means and standard deviations. However, it does not decorrelate the
activations due to the computationally costly matrix inversion. The batch normalizing transform is
as follows:
BN(h;
;) =+
hbE[h]q
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville