Our choice of 0.1 seems to work well across different tasks.
5 E XPERIMENTS
This section presents an empirical evaluation of the proposed batch-normalized LSTM on four different tasks. Note that for all the experiments, we initialize the batch normalization scale and shift
parameters
andto0:1and0respectively.
4
Published as a conference paper at ICLR 2017
0 20000 40000 60000 80000 100000
Training Iteration0.00.20.40.60.81.0AccuracyPixel-by-Pixel MNIST (Validation Set)
lstm
bn_lstm
0 20000 40000 60000 80000 100000
Training Iteration0.10.20.30.40.50.60.70.80.91.0AccuracyPixel-by-Pixel Permuted-MNIST (Validation Set)
lstm
bn_lstm
Figure 2: Accuracy on the validation set for the pixel by pixel MNIST classiﬁcation tasks. The
batch-normalized LSTM is able to converge faster relatively to a baseline LSTM. Batch-normalized
LSTM also shows some improve generalization on the permuted sequential MNIST that require to
preserve long-term memory information.
5.1 S EQUENTIAL MNIST
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville