section, we explain details on how to implement our model
and illustrate how we bypass the potential difﬁculties of
working in the complex domain.
4. Architecture details
In this section, we describe the nonlinearity we used, how
we incorporate real valued inputs with complex valued hidden units and map from complex hidden states to real outputs.
4.1. Complex hidden units
Our implementation represents all complex numbers using real values in terms of their real and imaginary parts.
Unitary Evolution Recurrent Neural Networks
Under this framework, we sidestep the lack of support
for complex numbers by most deep learning frameworks.
Consider multiplying the complex weight matrix W=
A+iBby the complex hidden vector h=x+iy,
where A;B;x;y are real. It is trivially true that Wh=
(AxBy) +i(Ay+Bx). When we represent v2Cnas
Re(v)>;Im(v)>>2R2n, we compute complex matrix
vector products with real numbers as follows
Re(Wh)
Im(Wh)
=AB
B ARe(h)
Im(h)
: (7)
More generally, let f:Cn!Cnbe any complex function andz=x+iyany complex vector. We may write
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio