Nair, Vinod and Hinton, Geoffrey E. Rectiﬁed linear units
improve restricted boltzmann machines. International
Conference on Machine Learning , 2010.
Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua.
On the difﬁculty of training recurrent neural networks.
International Conference on Machine Learning , 2010.
Saxe, Andrew M., McLelland, James L., and Ganguli,
Surya. Exact solutions to the nonlinear dynamics of
learning in deep linear neural networks. International
Conference in Learning Representations , 2014.
Tieleman, Tijmen and Hinton, Geoffrey. Lecture 6.5rmsprop: Divide the gradient by a running average of
its recent magnitude. Coursera: Neural Networks for
Machine Learning , 2012.
Yang, Zichao, Moczulski, Marcin, Denil, Misha, de Freitas, Nando, Smola, Alex, Song, Le, and Wang, Ziyu.
Deep fried convnets. International Conference on Computer Vision (ICCV) , 2015.
Zimmermann, Hans-Georg, Minin, Alexey, and
Kusherbaeva, Victoria. Comparison of the complex valued and real valued neural networks trained
with gradient descent and random search algorithms. In
ESANN , 2011.
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio