as easy to operate with. If further we were to use an invertible nonlinearity function, we would no longer need to store
hidden states, since they can be recomputed in the backward pass. This could have potentially huge implications,
as we would be able to reduce memory usage by an order
ofT, the number of time steps. This would make having
immensely large hidden layers possible, perhaps enabling
vast memory representations.
In this paper we demonstrate state of the art performance
on hard problems requiring long term reasoning and memory. These results are based on a novel parameterization
of unitary matrices which permit efﬁcient matrix computations and parameter optimization. Whilst complex domain modeling has been widely succesful in the signal processing community (e.g. Fourier transforms, wavelets), we
have yet to exploit the power of complex valued representation in the deep learning community. Our hope is that
this work will be a step forward in this direction. We motivate the idea of unitary evolution as a novel way to mitigate
the problems of vanishing and exploding gradients. Empirical evidence suggests that our uRNN is better able to pass
gradient information through long sequences and does not
suffer from saturating hidden states as much as LSTMs,
typical RNNs, or RNNs initialized with the identity weight
matrix (IRNNs).
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio