half. The output is the sum of the two entries of the ﬁrst sequence, corresponding to where the 1 entries are located in
the second sequence. A naive strategy of predicting 1 as the
output regardless of the input sequence gives an expected
mean squared error of 0:167, the variance of the sum of two
independent uniform distributions. This is our baseline to
beat.
Unitary Evolution Recurrent Neural Networks
Figure 3. Results on pixel by pixel MNIST classiﬁcation tasks. The uRNN is able to converge in a fraction of the iterations that the
LSTM requires. The LSTM performs better on MNIST classiﬁcation, but the uRNN outperforms on the more complicated task of
permuted pixels.
We chose to use 128 hidden units for the RNN with tanh,
IRNN and LSTM and 512 for the uRNN. This equates to
roughly 16K parameters for the RNN with tanh and IRNN,
60K for the LSTM and almost 9K for the uRNN. All models were trained using batch sizes of 20 and 50 with the best
results being reported. Our results are shown in Figure 2.
The LSTM and uRNN models are able to convincingly beat
the baseline up to T= 400 time steps. Both models do well
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio