batch of variable-length data with zeros. However, this biases the batch mean and variance of xt
toward zero. We address this effect using sequencewise normalization of the inputs as proposed
by Laurent et al. (2016); Amodei et al. (2015). That is, we share statistics over time for normalization
0 2000 4000 6000 8000 10000 12000 14000 16000
training steps1.41.61.82.02.22.4bits per characterLSTM
BN-LSTM
(a) Performance in bits-per-character on length100 subsequences of the Penn Treebank validation
sequence during training.
100 200 300 400 500 600 700 800 900 1000
sequence length1.321.341.361.381.401.421.441.46mean bits per characterLSTM
BN-LSTM, population statistics
BN-LSTM, batch statistics(b) Generalization to longer subsequences of Penn
Treebank using population statistics. The subsequences are taken from the test sequence.
Figure 3: Penn Treebank evaluation
7
Published as a conference paper at ICLR 2017
0 100 200 300 400 500 600 700 800
training steps (thousands)0.00.20.40.60.81.0error rateLSTM train
BN-LSTM train
BN-everywhere train
BN-e* train
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville