some information, it becomes much more difﬁcult to acquire further information pertaining to longer dependencies. We took the uRNN and LSTM models trained on
the adding problem with T= 200 , and computed a forward pass with newly generated data for the adding problem withT= 1000 . In order to show saturation effects,
we plot the norms of the hidden states and the L2distance
between each state and the last in Figure 4.
In our experiments, it is clear that the uRNN does not suffer
as much as other models do. Notice that whilst the norms of
hidden states in the uRNN grow very steadily over time, in
the LSTM they grow very fast, and then stay constant after
about 500time steps. This behaviour may suggest that the
LSTM hidden states saturate in their ability to incorporate
new information, which is vital for modeling long complicated sequences. It is interesting to see that the LSTM
hidden state at t= 500 , is close to that of t= 1000 , whilst
this is far from the case in the uRNN. Again, this suggests
that the LSTM’s capacity to use new information to alter
its hidden state severly degrades with sequence length. The
uRNN does not suffer from this difﬁculty nearly as badly.
A clear example of this phenomenon was observed in the
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio