beneﬁt over the baseline. The validation curves have minima of 50.3%, 49.5% and 50.0% for the
baseline, BN-LSTM and BN-everywhere respectively. We emphasize that these results were obtained without any tweaking – all we did was to introduce batch normalization.
BN-e* and BN-e** converge faster yet, and reach lower minima: 47.1% and 43.9% respectively.
Model CNN valid CNN test
Attentive Reader (Hermann et al., 2015) 38.4 37.0
LSTM (ours) 45.5 45.0
BN-e** (ours) 37.9 36.3
Table 4: Error rates on the CNN question-answering task Hermann et al. (2015).
We train and evaluate our best model, BN-e**, on the full task from (Hermann et al., 2015). On
this dataset we had to reduce the number of hidden units to 120 to avoid severe overﬁtting. Training
curves for BN-e** and a vanilla LSTM are shown in Figure 4(b). Table 4 reports performances of
the early-stopped models.
8
Published as a conference paper at ICLR 2017
6 C ONCLUSION
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville