RNNs are popular in sequence modeling thanks to their natural ability to process variable-length
sequences. However, training RNNs using ﬁrst-order stochastic gradient descent (SGD) is notoriously difﬁcult due to the well-known problem of exploding/vanishing gradients (Bengio et al., 1994;
Hochreiter, 1991; Pascanu et al., 2012). Gradient vanishing occurs when states htare not inﬂuenced
by small changes in much earlier states h,t, preventing learning of long-term dependencies in
the input data. Although learning long-term dependencies is fundamentally difﬁcult (Bengio et al.,
1994), its effects can be mitigated through architectural variations such as LSTM (Hochreiter &
Schmidhuber, 1997), GRU (Cho et al., 2014) and iRNN/uRNN (Le et al., 2015; Arjovsky et al.,
2015).
In what follows, we focus on the LSTM architecture (Hochreiter & Schmidhuber, 1997) with recurrent transition given by
0
BB@~ft
~it
~ot
~gt1
CCA=Whht1+Wxxt+b (2)
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville