gradient clipping at 1 to avoid exploding gradients.
The in-order MNIST task poses a unique problem for our model: the input for the ﬁrst hundred or so
timesteps is constant across examples since the upper pixels are almost always black. This causes the
variance of the hidden states to be exactly zero for a long period of time. Normalizing these zerovariance activations involves dividing zero by a small number at many timesteps, which does not
affect the forward-propagated activations but causes the back-propagated gradient to explode. We
work around this by adding Gaussian noise to the initial hidden states. Although the normalization
ampliﬁes the noise to signal level, we ﬁnd that it does not hurt performance compared to datadependent ways of initializing the hidden states.
Model MNIST pMNIST
TANH-RNN (Le et al., 2015) 35.0 35.0
iRNN (Le et al., 2015) 97.0 82.0
uRNN (Arjovsky et al., 2015) 95.1 91.4
sTANH-RNN (Zhang et al., 2016) 98.1 94.0
LSTM (ours) 98.9 90.2
BN-LSTM (ours) 99.0 95.4
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville