this task, beating the IRNN (Le et al., 2015), which reaches
close to 82% after 1 million training iterations. Notice that
uRNN reaches convergence in less than 20 thousand iterations, while it takes the LSTM from 5 to 10 times as many
to ﬁnish learning.
Permuting the pixels of MNIST images creates many
longer term dependencies across pixels than in the original pixel ordering, where a lot of structure is local. This
makes it necessary for a network to learn and remember
more complicated dependencies across varying time scales.
The results suggest that the uRNN is better able to deal
with such structure over the data, where the LSTM is better
suited to more local sequence structure tasks.
5.4. Exploratory experiments
Norms of hidden state gradients. As discussed in Section 2, key to being able to learn long term dependencies
is in controlling@C
@ht. With this in mind, we explored how
each model propagated gradients, by examining


@C
@ht


as
a function of t. Gradient norms were computed at the beUnitary Evolution Recurrent Neural Networks
Figure 4. From left to right. Norms of the gradients with respect to hidden states i.e.


@C
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio