memoryless strategy. This behaviour is consistent with the
results of (Graves et al., 2014), in which poor performance
is reported for the LSTM for a very similar long term memory problem.
The uRNN consistently achieves perfect performance inrelatively few iterations, even when having to recall sequences after 500 time steps. What is remarkable is that
the uRNN does not get stuck at the baseline at all, whilst
the LSTM and RNN do. This behaviour suggests that the
representations learned by the uRNN have qualitatively different properties from both the LSTM and classical RNNs.
5.2. Adding Problem
We closely follow the adding problem deﬁned in (Hochreiter & Schmidhuber, 1997) to explain the task at hand. Each
input consists of two sequences of length T. The ﬁrst sequence, which we denote x, consists of numbers sampled
uniformly at random U[0;1]. The second sequence is an indicator sequence consisting of exactly two entries of 1 and
remaining entries 0. The ﬁrst 1 entry is located uniformly
at random in the ﬁrst half of the sequence, whilst the second 1 entry is located uniformly at random in the second
half. The output is the sum of the two entries of the ﬁrst sequence, corresponding to where the 1 entries are located in
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio