gradients propagated back through a network vanish, the
credit assignment role of backpropagation is lost, as information about small changes in states in the far past has no
inﬂuence on future states. If gradients explode, gradientbased optimization algorithms struggle to traverse down a
cost surface, because gradient-based optimization assumes
small changes in parameters yield small changes in the objective function. As the number of time steps considered
in the sequence of states grows, the shrinking or expanding
effects associated with the state-to-state transformation at
individual time steps can grow exponentially, yielding respectively vanishing or exploding gradients. See Pascanu
et al. (2010) for a review.
Although the long-term dependencies problem appears
intractable in the absolute (Bengio et al., 1994) for
parametrized dynamical systems, several heuristics have
recently been found to help reduce its effect, such as the
use of self-loops and gating units in the LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Cho et al., 2014) recurrent architectures. Recent work also supports the idea
of using orthogonal weight matrices to assist optimization
(Saxe et al., 2014; Le et al., 2015).
In this paper, we explore the use of orthogonal and unitary
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio