important role in recurrent neural networks, which resemble very deep feed-forward networks.
Batch normalization (Ioffe & Szegedy, 2015) is a recently proposed technique for controlling the
distributions of feed-forward neural network activations, thereby reducing internal covariate shift. It
involves standardizing the activations going into each layer, enforcing their means and variances to
be invariant to changes in the parameters of the underlying layers. This effectively decouples each
layer’s parameters from those of other layers, leading to a better-conditioned optimization problem.
Indeed, deep neural networks trained with batch normalization converge signiﬁcantly faster and
generalize better.
Although batch normalization has demonstrated signiﬁcant training speed-ups and generalization
beneﬁts in feed-forward networks, it is proven to be difﬁcult to apply in recurrent architectures (Laurent et al., 2016; Amodei et al., 2015). It has found limited use in stacked RNNs, where the normalization is applied “vertically”, i.e. to the input of each RNN, but not “horizontally” between
timesteps. RNNs are deeper in the time direction, and as such batch normalization would be most
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville