Recently, Hermann et al. (2015) introduced a set of challenging benchmarks for natural language
processing, along with neural network architectures to address them. The tasks involve reading
real news articles and answering questions about their content. Their principal model, the Attentive Reader, is a recurrent neural network that invokes an attention mechanism to locate relevant
information in the document. Such models are notoriously hard to optimize and yet increasingly
popular.
To demonstrate the generality and practical applicability of our proposal, we apply batch normalization in the Attentive Reader model and show that this drastically improves training.
We evaluate several variants. The ﬁrst variant, referred to as BN-LSTM, consists of the vanilla Attentive Reader model with the LSTM simply replaced by our BN-LSTM reparameterization. The
second variant, termed BN-everywhere, is exactly like the ﬁrst, except that we also introduce batch
normalization into the attention computations, normalizing each term going into the tanh nonlinearities.
Our third variant, BN-e*, is like BN-everywhere, but improved to more carefully handle variablelength sequences. Throughout this experiment we followed the common practice of padding each
batch of variable-length data with zeros. However, this biases the batch mean and variance of xt
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville