gamma=0.40
gamma=0.50
gamma=0.60
gamma=0.70
gamma=0.80
gamma=0.90
gamma=1.00
(a) We visualize the gradient ﬂow through a batchnormalized tanh RNN as a function of 
. High
variance causes vanishing gradient.
0.0 0.2 0.4 0.6 0.8 1.0
input standard deviation0.00.20.40.60.81.0expected derivative (and IQR range)derivative through tanh(b) We show the empirical expected derivative and
interquartile range of tanh nonlinearity as a function of input variance. High variance causes saturation, which decreases the expected derivative.
Figure 1: Inﬂuence of pre-activation variance on gradient propagation.
In Figure 1(a), we show how the pre-activation variance impacts gradient propagation in a simple
RNN on the sequential MNIST task described in Section 5.1. Since backpropagation operates in
reverse, the plot is best read from right to left. The quantity plotted is the norm of the gradient
of the loss with respect to the hidden state at different time steps. For large values of 
, the norm
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville