BN-LSTM train
BN-everywhere train
BN-e* train
BN-e** train
LSTM valid
BN-LSTM valid
BN-everywhere valid
BN-e* valid
BN-e** valid
(a) Error rate on the validation set for the Attentive Reader models on a variant of the CNN QA
task (Hermann et al., 2015). As detailed in Appendix C, the theoretical lower bound on the error
rate on this task is 43%.
0 50 100 150 200 250 300 350 400
training steps (thousands)0.10.20.30.40.50.60.70.80.91.0error rateLSTM train
BN-e** train
LSTM valid
BN-e** valid(b) Error rate on the validation set on the full CNN
QA task from Hermann et al. (2015).
Figure 4: Training curves on the CNN question-answering tasks.
of the input terms Wxxt, but notfor the recurrent terms Whhtor the cell output ct. Doing so avoids
many issues involving degenerate statistics due to input sequence padding.
Our fourth and ﬁnal variant BN-e** is like BN-e* but bidirectional. The main difﬁculty in adapting
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville