training, we segment the training sequence into examples of length 100. The training sequence does
not cleanly divide by 100, so for each epoch we randomly crop a subsequence that does and segment
that instead.
Our baseline is an LSTM with 1000 units, trained to predict the next character using a softmax
classiﬁer on the hidden state ht. We use stochastic gradient descent on minibatches of size 64, with
gradient clipping at 1.0 and step rule determined by Adam (Kingma & Ba, 2014) with learning rate
0.002. We use orthogonal initialization for all weight matrices. The setup for the batch-normalized
LSTM is the same in all respects except for the introduction of batch normalization as detailed in 3.
We show the learning curves in Figure 3(a). BN-LSTM converges faster and generalizes better than
the LSTM baseline. Figure 3(b) shows the generalization of our model to longer sequences. We
observe that using the population statistics improves generalization performance, which conﬁrms
that repeating the last population statistic (cf. Section 3) is a viable strategy. In table 2 we report
the performance of our best models (early-stopped on validation performance) on the Penn Treebank
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville