been mathematically proven to avoid exploding gradients.
3. Unitary Evolution RNNs
Unitary matrices generalize orthogonal matrices to the
complex domain. A complex valued, norm preserving matrix,U, is called a unitary matrix and is such that UU=
UU=I, where Uis the conjugate transpose of U. Directly parametrizing the set of unitary matrices in such a
way that gradient-based optimization can be applied is not
straightforward because a gradient step will typically yield
a matrix that is not unitary, and projecting on the set of unitary matrices (e.g., by performing an eigendecomposition)
generally costsO(n3)computation when Uisnn.
The most important feature of unitary and orthogonal matrices for our purpose is that they have eigenvalues jwith
Unitary Evolution Recurrent Neural Networks
absolute value 1. The following lemma, proved in (Hoffman & Kunze, 1971), may shed light on a method which
can be used to efﬁciently span a large set of unitary matrices.
Lemma 1. A complex square matrix Wis unitary if and
only if it has an eigendecomposition of the form W=
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio