complexity measures of recurrent neural networks. arXiv:1602.08210 , 2016.
10
Published as a conference paper at ICLR 2017
A C ONVERGENCE OF POPULATION STATISTICS
0.20
0.15
0.10
0.05
0.000.050.100.150.20mean of recurrent term
0.20
0.15
0.10
0.05
0.000.050.100.150.20mean of cell state
0 10 20 30 40 50
time steps0.00.51.01.52.0variance of recurrent term
0 10 20 30 40 50
time steps0.0010.0020.0030.0040.0050.0060.0070.0080.009variance of cell state
Figure 5: Convergence of population statistics to stationary distributions on the Penn Treebank task.
The horizontal axis denotes RNN time. Each curve corresponds to a single hidden unit. Only a
random subset of units is shown. See Section 3 for discussion.
B S ENSITIVITY TO INITIALIZATION OF 
In Section 4 we investigated the effect of initial 
on gradient ﬂow. To show the practical implications of this, we performed several experiments on the pMNIST and Penn Treebank benchmarks.
The resulting performances are shown in Figure 6.
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville