of the loss with respect to the hidden state at different time steps. For large values of 
, the norm
quickly goes to zero as gradient is propagated back in time. For small values of 
the norm is nearly
constant.
To demonstrate what we think is the cause of this vanishing, we drew samples xfrom a set of
centered Gaussian distributions with standard deviation ranging from 0 to 1, and computed the
derivative tanh0(x) = 1tanh2(x)2[0;1]for each. Figure 1(b) shows the empirical distribution
of the derivative as a function of standard deviation. When the input standard deviation is low, the
input tends to be close to the origin where the derivative is close to 1. As the standard deviation
increases, the expected derivative decreases as the input is more likely to be in the saturation regime.
At unit standard deviation, the expected derivative is much smaller than 1.
We conjecture that this is what causes the gradient to vanish, and recommend initializing 
to a
small value. In our trials we found that values of 0.01 or lower caused instabilities during training.
Our choice of 0.1 seems to work well across different tasks.
5 E XPERIMENTS
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville