& Ba, 2014) with learning rate 0.001. All weight matrices were initialized to be orthogonal.
6
Published as a conference paper at ICLR 2017
We early-stop on validation performance and report the test performance of the resulting model in
table 3. We observe that BN-LSTM obtains a signiﬁcant performance improvement over the LSTM
baseline. Chung et al. (2016) has since improved on our performance.
Model text8
td-LSTM (Zhang et al., 2016) 1.63
HF-MRNN (Mikolov et al., 2012) 1.54
skipping RNN (Pachitariu & Sahani, 2013) 1.48
LSTM (ours) 1.43
BN-LSTM (ours) 1.36
HM-LSTM (Chung et al., 2016) 1.29
Table 3: Bits-per-character on the text8 test sequence.
5.4 T EACHING MACHINES TO READ AND COMPREHEND
Recently, Hermann et al. (2015) introduced a set of challenging benchmarks for natural language
processing, along with neural network architectures to address them. The tasks involve reading
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville