sequence data. (Bengio et al., 1994) suggest that having
a large memory may be crucial for solving difﬁcult tasks
with long ranging dependencies: the smaller the state dimension, the more information necessarily has to be eliminated when mapping a long sequence to a ﬁxed-dimension
state.
We call any RNN architecture which uses a unitary hidden
to hidden matrix a unitary evolution RNN (uRNN). After
experimenting with several structures, we settled on the following composition
W=D3R2F1D2R 1FD1: (6)
Whilst each but the permutation matrix is complex, we
parameterize and represent them with real numbers for implementation purposes. When the ﬁnal cost is real and differentiable, we may perform gradient descent optimization
to learn the parameters. (Yang et al., 2015) construct a real
valued, non-orthogonal matrix using a similar parameterization with the motivation of parameter reduction by an
order of magnitude on an industrial sized network. This
combined with earlier work (Le et al., 2010) suggests that it
is possible to create highly expressive matrices by composing simple matrices with few parameters. In the following
section, we explain details on how to implement our model
and illustrate how we bypass the potential difﬁculties of
working in the complex domain.
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio