gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70
gamma 1.00
0 5000 10000 15000
training steps1.001.021.041.061.081.10bits per characterPTB valid
gamma 0.10
gamma 0.30
gamma 0.50
gamma 0.70
gamma 1.00
Figure 6: Training curves on pMNIST and Penn Treebank for various initializations of 
.
answers from the passage, putting an upper bound of 57% on the validation accuracy that can be
achieved.
For the reported performances, the ﬁrst three models (LSTM, BN-LSTM and BN-everywhere) are
trained using the exact same hyperparameters, which were chosen because they work well for the
baseline. The hidden state is composed of 240 units. We use stochastic gradient descent on minibatches of size 64, with gradient clipping at 10 and step rule determined by Adam (Kingma & Ba,
2014) with learning rate 8105.
For BN-e* and BN-e**, we use the same hyperparameters except that we reduce the learning rate to
8104and the minibatch size to 40.
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville