LSTM also shows some improve generalization on the permuted sequential MNIST that require to
preserve long-term memory information.
5.1 S EQUENTIAL MNIST
We evaluate our batch-normalized LSTM on a sequential version of the MNIST classiﬁcation
task (Le et al., 2015). The model processes each image one pixel at a time and ﬁnally predicts
the label. We consider both sequential MNIST tasks, MNIST and permuted MNIST ( pMNIST). In
MNIST, the pixels are processed in scanline order. In pMNIST the pixels are processed in a ﬁxed
random order.
Our baseline consists of an LSTM with 100 hidden units, with a softmax classiﬁer to produce a
prediction from the ﬁnal hidden state. We use orthogonal initialization for all weight matrices,
except for the hidden-to-hidden weight matrix which we initialize to be the identity matrix, as this
yields better generalization performance on this task for both models. The model is trained using
RMSProp (Tieleman & Hinton, 2012) with learning rate of 103and0:9momentum. We apply
gradient clipping at 1 to avoid exploding gradients.
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville