S. Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Master’s thesis , 1991.
S. Hochreiter and J Schmidhuber. Long short-term memory. Neural computation , 1997.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. abs/1502.03167, 2015.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv:1412.6980 , 2014.
D Krueger and R. Memisevic. Regularizing rnns by stabilizing activations. ICLR , 2016.
9
Published as a conference paper at ICLR 2017
David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, and Aaron Courville. Zoneout:
Regularizing rnns by randomly preserving hidden activations. arXiv:1606.01305 , 2016.
C. Laurent, G. Pereyra, P. Brakel, Y . Zhang, and Y . Bengio. Batch normalized recurrent neural
networks. ICASSP , 2016.
We propose a reparameterization of LSTM that brings the benefits of batch
normalization to recurrent neural networks. Whereas previous works only apply
batch normalization to the input-to-hidden transformation of RNNs, we
demonstrate that it is both possible and beneficial to batch-normalize the
hidden-to-hidden transition, thereby reducing internal covariate shift between
time steps. We evaluate our proposal on various sequential problems such as
sequence classification, language modeling and question answering. Our
empirical results show that our batch-normalized LSTM consistently leads to
faster convergence and improved generalization.
Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville