parameters, one per dimension. Note that the modReLU
is similar to the ReLU in spirit, in fact more concretely
modReLU (z) =ReLU (jzj+b)z
jzj.To map hidden states to output, we deﬁne a matrix U2
Rno2nh, wherenois the output dimension. We calculate
a linear output as
ot=URe(ht)
Im(ht)
+bo; (9)
wherebo2Rnois the output bias. The linear output is real
valued (ot2Rno) and can be used for prediction and loss
function calculation akin to typical neural networks (e.g. it
may be passed through a softmax which is used for cross
entropy calculation for classiﬁcation tasks).
4.3. Initialization
Due to the stability of the norm preserving operations of
our network, we found that performance was not very sensitive to initialization of parameters. For full disclosure
and reproducibility, we explain our initialization strategy
for each parameter below.
We initialize VandU(the input and output matrices) as in (Glorot & Bengio, 2010),
with weights sampled independently from uniforms,
Uh
p
6pnin+nout;p
6pnin+nouti
.
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio