Hidden state size: 60, 120, 240, 280
Table 5: Hyperparameter values that have been explored in the experiments.
For MNIST and pMNIST, the hyperparameters were varied independently. For Penn Treebank, we
performed a full grid search on learning rate and hidden state size, and later performed a sensitivity
12
Published as a conference paper at ICLR 2017
analysis on the batch size and initial 
. For the text8 task and the experiments with the Attentive
Reader, we carried out a grid search on the learning rate and hidden state size.
The same values were tried for both the baseline and our BN-LSTM. In each case, our reported
results are those of the model with the best validation performance.
13
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville