length. The LSTM and the uRNN show similar performance across time lengths, consistently beating the baseline.
sequence in exactly the same order. The goal is to minimize the average cross entropy of category predictions at
each time step of the sequence. The task amounts to having to remember a categorical sequence of length 10, for T
time steps.
A simple baseline can be established by considering an
optimal strategy when no memory is available, which we
deem the memoryless strategy. The memoryless strategy
would be to predict a8forT+ 10 entries and then predict
each of the ﬁnal 10categories from the set faig7
i=0independently and uniformly at random. The categorical cross
entropy of this strategy is10 log(8)
T+20.
We ran experiments where the RNN with tanh activations,
IRNN, LSTM and uRNN had hidden layers of size 80, 80,
40 and 128 respectively. This equates to roughly 6500 parameters per model. In Figure 1, we see that aside from the
simplest case, both the RNN with tanh and more surprisingly the LSTMs get almost exactly the same cost as the
memoryless strategy. This behaviour is consistent with the
results of (Graves et al., 2014), in which poor performance
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio