memory (Hochreiter & Schmidhuber, 1997; Le et al., 2015;
Graves et al., 2014; Martens & Sutskever, 2011)
Of the handful of optimization algorithms we tried on the
various models, RMSProp (Tieleman & Hinton, 2012) lead
to fastest convergence and is what we stuck to for all experiments here on in. However, we found the IRNN to be
particularly unstable; it only ran without blowing up with
incredibly low learning rates and gradient clipping. Since
the performance was so poor relative to other models we
compare against, we do not show IRNN curves in the ﬁgures. In each experiment we use a learning rate of 103
and a decay rate of 0:9. For the LSTM and RNN models,
we had to clip gradients at 1 to avoid exploding gradients.Gradient clipping was unnecessary for the uRNN.
5.1. Copying memory problem
Recurrent networks have been known to have trouble remembering information about inputs seen many time steps
previously (Bengio et al., 1994; Pascanu et al., 2010). We
therefore want to test the uRNN’s ability to recall exactly
data seen a long time ago.
Following a similar setup to (Hochreiter & Schmidhuber,
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio