the performance of our best models (early-stopped on validation performance) on the Penn Treebank
test sequence. Follow up works havd since improved the state of the art (Krueger et al., 2016; Chung
et al., 2016; Ha et al., 2016).
5.3 T EXT8
We evaluate our model on a second character-level language modeling task on the much larger
text8 dataset (Mahoney, 2009). This dataset is derived from Wikipedia and consists of a sequence
of 100M characters including only alphabetical characters and spaces. We follow Mikolov et al.
(2012); Zhang et al. (2016) and use the ﬁrst 90M characters for training, the next 5M for validation
and the ﬁnal 5M characters for testing. We train on nonoverlapping sequences of length 180.
Both our baseline and batch-normalized models are LSTMs with 2000 units, trained to predict the
next character using a softmax classiﬁer on the hidden state ht. We use stochastic gradient descent
on minibatches of size 128, with gradient clipping at 1.0 and step rule determined by Adam (Kingma
& Ba, 2014) with learning rate 0.001. All weight matrices were initialized to be orthogonal.
6
Published as a conference paper at ICLR 2017
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville