transform BN(;
;)into the LSTM as follows:
0
BB@~ft
~it
~ot
~gt1
CCA= BN( Whht1;
h;h) + BN( Wxxt;
x;x) +b (6)
ct=(~ft)ct1+(~it)tanh( ~gt) (7)
ht=(~ot)tanh(BN( ct;
c;c)) (8)
In our formulation, we normalize the recurrent term Whht1and the input term Wxxtseparately.
Normalizing these terms individually gives the model better control over the relative contribution
of the terms using the 
hand
xparameters. We set h=x=0to avoid unnecessary redundancy, instead relying on the pre-existing parameter vector bto account for both biases. In order to
leave the LSTM dynamics intact and preserve the gradient ﬂow through ct, we do not apply batch
normalization in the cell update.
The batch normalization transform relies on batch statistics to standardize the LSTM activations. It
would seem natural to share the statistics that are used for normalization across time, just as recurrent
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville