8104and the minibatch size to 40.
D H YPERPARAMETER SEARCHES
Table 5 reports hyperparameter values that were tried in the experiments.
(a) MNIST and pMNIST
Learning rate: 1e-2, 1e-3, 1e-4
RMSProp momentum: 0.5, 0.9
Hidden state size: 100, 200, 400
Initial
: 1e-1, 3e-1, 5e-1, 7e-1, 1.0(b) Penn Treebank
Learning rate: 1e-1, 1e-2, 2e-2, 1e-3
Hidden state size: 800, 1000, 1200, 1500, 2000
Batch size: 32, 64, 100, 128
Initial
: 1e-1, 3e-1, 5e-1, 7e-1, 1.0
(c) Text8
Learning rate: 1e-1, 1e-2, 1e-3
Hidden state size: 500, 1000, 2000, 4000(d) Attentive Reader
Learning rate: 8e-3, 8e-4, 8e-5, 8e-6
Hidden state size: 60, 120, 240, 280
Table 5: Hyperparameter values that have been explored in the experiments.
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville