Whilst our model uses complex valued matrices and parameters, all implementation and optimization is possible
with real numbers and has been done in Theano (Bergstra
et al., 2010). This along with other implementation details
are discussed in Section 4, and the code used for the experiments is available online. The potential of the developed
model for learning long term dependencies with relatively
few parameters is explored in Section 5. We ﬁnd that the
proposed architecture generally outperforms LSTMs and
previous approaches based on orthogonal initialization.
2. Orthogonal Weights and Bounding the
Long-Term Gradient
A matrix, W, is orthogonal if W>W=WW>=I.
Orthogonal matrices have the property that they preserve
norm (i.e.kWhk2=khk2) and hence repeated iterative
multiplication of a vector by an orthogonal matrix leaves
the norm of the vector unchanged.
LethTandhtbe the hidden unit vectors for hidden layers
Tandtof a neural network with Thidden layers and T
t. IfCis the objective we are trying to minimize, then
thevanishing andexploding gradient problems refer to the
decay or growth of@C
@htas the number of layers, T, grows.
Letbe a pointwise nonlinearity function, and
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio