The resulting performances are shown in Figure 6.
ThepMNIST training curves conﬁrm that higher initial values of 
are detrimental to the optimization of the model. For the Penn Treebank task however, the effect is gone.
We believe this is explained by the difference in the nature of the two tasks. For pMNIST, the model
absorbs the input sequence and only at the end of the sequence does it make a prediction on which it
receives feedback. Learning from this feedback requires propagating the gradient all the way back
through the sequence.
In the Penn Treebank task on the other hand, the model makes a prediction at each timestep. At
each step of the backward pass, a fresh learning signal is added to the backpropagated gradient.
Essentially, the model is able to get off the ground by picking up short-term dependencies. This fails
onpMNIST wich is dominated by long-term dependencies (Arjovsky et al., 2015).
C T EACHING MACHINES TO READ AND COMPREHEND : TASK SETUP
We evaluate the models on the question answering task using the CNN corpus (Hermann et al.,
2015), with placeholders for the named entities. We follow a similar preprocessing pipeline as Hermann et al. (2015). During training, we randomly sample the examples with replacement and shufﬂe
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville