D, a diagonal matrix with Dj;j=eiwj, with parameterswj2R,
R=I2vv
kvk2, a reﬂection matrix in the complex
vectorv2Cn,
, a ﬁxed random index permutation matrix, and
 F andF1, the Fourier and inverse Fourier transforms.Appealingly, D,Randall permitO(n)storage and
O(n)computation for matrix vector products. FandF1
require no storage and O(nlogn)matrix vector multiplication using the Fast Fourier Transform algorithm. A major
advantage of composing unitary matrices of the form listed
above, is that the number of parameters, memory and computational cost increase almost linearly in the size of the
hidden layer. With such a weight matrix, immensely large
hidden layers are feasible to train, whilst being impossible
in traditional neural networks.
With this in mind, in this work we choose to consider recurrent neural networks with unitary hidden to hidden weight
matrices. Our claim is that the ability to have large hidden
layers where hidden states norms are preserved provides
a powerful tool for modeling long term dependencies in
sequence data. (Bengio et al., 1994) suggest that having
a large memory may be crucial for solving difﬁcult tasks
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio