In this task, suggested by (Le et al., 2015), algorithms are
fed pixels of MNIST (LeCun et al., 1998) sequentially and
required to output a class label at the end. We consider two
tasks: one where pixels are read in order (from left to right,
bottom to top) and one where the pixels are all randomly
permuted using the same randomly generated permutation
matrix. The same model architectures as for the addingproblem were used for this task, except we now use a softmax for category classiﬁcation. We ran the optimization
algorithms until convergence of the mean categorical cross
entropy on test data, and plot test accuracy in Figure 3.
Both the uRNN and LSTM perform applaudably well here.
On the correct unpermuted MNIST pixels, the LSTM performs better, achieving 98.2 % test accurracy versus 95.1%
for the uRNN. However, when we permute the ordering of
the pixels, the uRNN dominates with 91.4% of accuracy in
contrast to the 88% of the LSTM, despite having less than
a quarter of the parameters. This result is state of the art on
this task, beating the IRNN (Le et al., 2015), which reaches
close to 82% after 1 million training iterations. Notice that
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio