Our fourth and ﬁnal variant BN-e** is like BN-e* but bidirectional. The main difﬁculty in adapting
to bidirectional models also involves padding. Padding poses no problem as long as it is properly
ignored (by not updating the hidden states based on padded regions of the input). However to
perform the reverse application of a bidirectional model, it is common to simply reverse the padded
sequences, thus moving the padding to the front. This causes similar problems as were observed
on the sequential MNIST task (Section 5.1): the hidden states will not diverge during the initial
timesteps and hence their variance will be severely underestimated. To get around this, we reverse
only the unpadded portion of the input sequences and leave the padding in place.
See Appendix C for hyperparameters and task details.
Figure 4(a) shows the learning curves for the different variants of the attentive reader. BN-LSTM
trains dramatically faster than the LSTM baseline. BN-everywhere in turn shows a signiﬁcant improvement over BN-LSTM. In addition, both BN-LSTM and BN-everywhere show a generalization
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville