the order of the placeholders in each text inside the minibatch. We use a vocabulary of 65829 words.
We deviate from Hermann et al. (2015) in order to save computation: we use only the 4 most relevant
sentences from the description, as identiﬁed by a string matching procedure. Both the training and
validation sets are preprocessed in this way. Due to imprecision this heuristic sometimes strips the
11
Published as a conference paper at ICLR 2017
0 10000 20000 30000 40000 50000
training steps0.00.51.01.52.02.5cross entropyPermuted MNIST train
gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70
gamma 1.00
0 10000 20000 30000 40000 50000
training steps0.00.51.01.52.02.5cross entropyPermuted MNIST valid
gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70
gamma 1.00
0 5000 10000 15000
training steps0.800.850.900.951.001.051.10bits per characterPTB train
gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville