expressive unitary weight matrix by composing
several structured matrices that act as building
blocks with parameters to be learned. Optimization with this parameterization becomes feasible
only when considering hidden states in the complex domain. We demonstrate the potential of
this architecture by achieving state of the art results in several hard tasks involving very longterm dependencies.
1. Introduction
Deep Neural Networks have shown remarkably good performance on a wide range of complex data problems including speech recognition (Hinton et al., 2012), image
recognition (Krizhevsky et al., 2012) and natural language
processing (Collobert et al., 2011). However, training very
deep models remains a difﬁcult task. The main issue surrounding the training of deep networks is the vanishing
andexploding gradients problems introduced by HochreProceedings of the 33rdInternational Conference on Machine
Learning , New York, NY , USA, 2016. JMLR: W&CP volume
48. Copyright 2016 by the author(s).iter (1991) and shown by Bengio et al. (1994) to be necessarily arising when trying to learn to reliably store bits
of information in any parametrized dynamical system. If
gradients propagated back through a network vanish, the
credit assignment role of backpropagation is lost, as information about small changes in states in the far past has no
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio