uRNN does not suffer from this difﬁculty nearly as badly.
A clear example of this phenomenon was observed in the
adding problem with T= 750 . We found that the Pearson
correlation between the LSTM output prediction and the
ﬁrst of the two uniform samples (whose sum is the target
output) was= 0:991. This suggests that the LSTM learnt
to simply ﬁnd and store the ﬁrst sample, as it was unable
to incorporate any more information by the time it reachedthe second, due to saturation of the hidden states.
6. Discussion
There are a plethora of further ideas that may be explored
from our ﬁndings, both with regards to learning representation and efﬁcient implementation. For example, one hurdle
of modeling long sequences with recurrent networks is the
requirement of storing all hidden state values for the purpose of gradient backpropagation. This can be prohibitive,
since GPU memory is typically a limiting factor of neural
network optimization. However, since our weight matrix is
unitary, its inverse is its conjugate transpose, which is just
as easy to operate with. If further we were to use an invertible nonlinearity function, we would no longer need to store
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio