with weights sampled independently from uniforms,
Uh
p
6pnin+nout;p
6pnin+nouti
.
The biases,bandboare initialized to 0. This implies
that at initialization, the network is linear with unitary
weights, which seems to help early optimization (Saxe
et al., 2014).
The reﬂection vectors for R1andR2are initialized
coordinate-wise from a uniform U[1;1]. Note that
the reﬂection matrices are invariant to scalar multiplication of the parameter vector, hence the width of the
uniform initialization is unimportant.
The diagonal weights for D1;D2andD3are sampled from a uniform, U[;]. This ensures that the
diagonal entries Dj;jare sampled uniformly over the
complex unit circle.
We initialize h0with a uniform,Uh
q
3
2nh;q
3
2nhi
,
which results in E
kh0k2
= 1. Since the norm of
the hidden units are roughly preserved through unitary
evolution and inputs are typically whitened to have
norm 1, we have hidden states, inputs and linear outputs of the same order of magnitude, which seems to
help optimization.
5. Experiments
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio