LSTM (ours) 98.9 90.2
BN-LSTM (ours) 99.0 95.4
Table 1: Accuracy obtained on the test set for the pixel by pixel MNIST classiﬁcation tasks
In Figure 2 we show the validation accuracy while training for both LSTM and batch-normalized
LSTM (BN-LSTM). BN-LSTM converges faster than LSTM on both tasks. Additionally, we observe that BN-LSTM generalizes signiﬁcantly better on pMNIST. It has been highlighted in Arjovsky et al. (2015) that pMNIST contains many longer term dependencies across pixels than in
the original pixel ordering, where a lot of structure is local. A recurrent network therefore needs to
5
Published as a conference paper at ICLR 2017
Model Penn Treebank
LSTM (Graves, 2013) 1.262
HF-MRNN (Mikolov et al., 2012) 1.41
Norm-stabilized LSTM (Krueger & Memisevic, 2016) 1.39
ME n-gram (Mikolov et al., 2012) 1.37
LSTM (ours) 1.38
BN-LSTM (ours) 1.32
Zoneout (Krueger et al., 2016) 1.27
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville