LSTM (ours) 1.38
BN-LSTM (ours) 1.32
Zoneout (Krueger et al., 2016) 1.27
HM-LSTM (Chung et al., 2016) 1.24
HyperNetworks (Ha et al., 2016) 1.22
Table 2: Bits-per-character on the Penn Treebank test sequence.
characterize dependencies across varying time scales in order to solve this task. Our results suggest
that BN-LSTM is better able to capture these long-term dependencies.
Table 1 reports the test set accuracy of the early stop model for LSTM and BN-LSTM using the population statistics. Recurrent batch normalization leads to a better test score, especially for pMNIST
where models have to leverage long-term temporal depencies. In addition, Table 1 shows that our
batch-normalized LSTM achieves state of the art on both MNIST and pMNIST.
5.2 C HARACTER -LEVEL PENN TREEBANK
We evaluate our model on the task of character-level language modeling on the Penn Treebank
corpus (Marcus et al., 1993) according to the train/valid/test partition of Mikolov et al. (2012). For
training, we segment the training sequence into examples of length 100. The training sequence does
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville