(Saxe et al., 2014; Le et al., 2015).
In this paper, we explore the use of orthogonal and unitary
matrices in recurrent neural networks. We start in Section 2
by showing a novel bound on the propagated gradients in
recurrent nets when the recurrent matrix is orthogonal. Section 3 discusses the difﬁculties of parameterizing real valued orthogonal matrices and how they can be alleviated by
moving to the complex domain.
We discuss a novel approach to constructing expressive
unitary matrices as the composition of simple unitary matrices which require at most O(nlogn)computation and
O(n)memory, when the state vector has dimension n.
These are unlike general matrices, which require O(n2)
computation and memory. Complex valued representationsarXiv:1511.06464v4  [cs.LG]  25 May 2016
Unitary Evolution Recurrent Neural Networks
have been considered for neural networks in the past, but
with limited success and adoption (Hirose, 2003; Zimmermann et al., 2011). We hope our ﬁndings will change this.
Whilst our model uses complex valued matrices and parameters, all implementation and optimization is possible
with real numbers and has been done in Theano (Bergstra
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio