The LSTM and uRNN models are able to convincingly beat
the baseline up to T= 400 time steps. Both models do well
whenT= 750 , but the mean squared error does not reach
close to 0. The uRNN achieves lower test error, but it’s
curve is more noisy. Despite having vastly more parameters, we monitored the LSTM performance to ensure no
overﬁtting.
The RNN with tanh and IRNN were not able to beat the
baseline for any number of time steps. (Le et al., 2015) report that their RNN solve the problem for T= 150 and the
IRNN forT= 300 , but they require over a million iterations before they start learning. Neither of the two models came close to either the uRNN or the LSTM in performance. The stark difference in our ﬁndings are best explained by our use of RMSprop with signiﬁcantly higher
learning rates ( 103as opposed to 108) than (Le et al.,
2015) use for SGD with momentum.
5.3. Pixel-by-pixel MNIST
In this task, suggested by (Le et al., 2015), algorithms are
fed pixels of MNIST (LeCun et al., 1998) sequentially and
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio