data seen a long time ago.
Following a similar setup to (Hochreiter & Schmidhuber,
1997), we outline the copy memory task. Consider 10 categories,faig9
i=0. The input takes the form of a T+20 length
vector of categories, where we test over a range of values
ofT. The ﬁrst 10entries are sampled uniformly, independently and with replacement from faig7
i=0, and represent
the sequence which will need to be remembered. The next
T1entries are set to a8, which can be thought of as the
’blank’ category. The next single entry is a9, which represents a delimiter, which should indicate to the algorithm
that it is now required to reproduce the initial 10categories
in the output. The remaining 10entries are set to a8. The
required output sequence consists of T+ 10 repeated entries ofa8, followed by the ﬁrst 10categories of the input
Unitary Evolution Recurrent Neural Networks
Figure 2. Results of the adding problem for T= 100 ;200;400;750. The RNN with tanh is not able to beat the baseline for any time
length. The LSTM and the uRNN show similar performance across time lengths, consistently beating the baseline.
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio