as follows:
BN(h;
;) =+
hbE[h]q
dVar[h] +(5)
where h2Rdis the vector of (pre)activations to be normalized, 
2Rd;2Rdare model
parameters that determine the mean and standard deviation of the normalized activation, and 2R
is a regularization hyperparameter. The division should be understood to proceed elementwise.
At training time, the statistics E[h]andVar[h]are estimated by the sample mean and sample variance of the current minibatch. This allows for backpropagation through the statistics, preserving the
convergence properties of stochastic gradient descent. During inference, the statistics are typically
estimated based on the entire training set, so as to produce a deterministic prediction.
3 B ATCH -NORMALIZED LSTM
This section introduces a reparameterization of LSTM that takes advantage of batch normalization.
Contrary to Laurent et al. (2016); Amodei et al. (2015), we leverage batch normalization in both
the input-to-hidden andthe hidden-to-hidden transformations. We introduce the batch-normalizing
transform BN(;
;)into the LSTM as follows:
0
BB@~ft
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville