matrix, W. We must (i) ensure the columns of Vcome
in complex conjugate pairs, vk=vj, and (ii) tie weights
wk=wjin order to achieve eiwj=eiwk. Most neural network objective functions are differentiable with respect to the weight matrices, and consequently wjmay be
learned by gradient descent.
Unfortunately the above approach has undesirable properties. Fixing Vand learning wrequiresO
n2
memory,
which is unacceptable given that the number of learned parameters isO(n). Further note that calculating Vufor an
arbitrary vector urequiresO(n2)computation. Setting V
to the identity would satisfy the conditions of the lemma,
whilst reducing memory and computation requirements to
O(n), however, Wwould remain diagonal, and have poor
representation capacity.
We propose an alternative strategy to parameterize unitary
matrices. Since the product of unitary matrices is itself a
unitary matrix, we compose several simple, parameteric,
unitary matrices to construct a single, expressive unitary
matrix. The four unitary building blocks considered are
D, a diagonal matrix with Dj;j=eiwj, with parameterswj2R,
R=I2vv
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio