norm 1, we have hidden states, inputs and linear outputs of the same order of magnitude, which seems to
help optimization.
5. Experiments
In this section we explore the performance of our uRNN
in relation to (a) RNN with tanh activations, (b) IRNN (Le
et al., 2015), that is an RNN with ReLU activations and
with the recurrent weight matrix initialized to the identity,
Unitary Evolution Recurrent Neural Networks
Figure 1. Results of the copying memory problem for time lags of 100;200;300;500. The LSTM is able to beat the baseline only for
100times steps. Conversely the uRNN is able to completely solve each time length in very few training iterations, without getting stuck
at the baseline.
and (c) LSTM (Hochreiter & Schmidhuber, 1997) models. We show that the uRNN shines quantitatively when
it comes to modeling long term dependencies and exhibits
qualitatively different learning properties to the other models.
We chose a handful of tasks to evaluate the performance
of the various models. The tasks were especially created
to be be pathologically hard, and have been used as benchmarks for testing the ability of a model to capture long-term
memory (Hochreiter & Schmidhuber, 1997; Le et al., 2015;
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio