Figure 4. From left to right. Norms of the gradients with respect to hidden states i.e.


@C
@ht


at (i) beginning of training, (ii) after 100
iterations. (iii) Norms of the hidden states and (iv) L2distance between hidden states and ﬁnal hidden state. The gradient norms of
uRNNs do not decay as fast as for other models as training progresses. uRNN hidden state norms stay much more consistent over time
than the LSTM. LSTM hidden states stay almost the same after a number of time steps, suggesting that it is not able to use new input
information.
ginning of training and again after 100 iterations of training
on the adding problem. The curves are plotted in Figure 4.
It is clear that at ﬁrst, the uRNN propagates gradients perfectly, while each other model has exponentially vanishing
gradients. After 100 iterations of training, each model experiences vanishing gradients, but the uRNN is best able to
propagate information, having much less decay.
Hidden state saturation. We claim that typical recurrent
architectures saturate, in the sense that after they acquire
some information, it becomes much more difﬁcult to acquire further information pertaining to longer dependencies. We took the uRNN and LSTM models trained on
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio