ReLU is a natural choice in combination with a norm preserving weight matrix. We ﬁrst experimented with placing
separate ReLU activations on the real and imaginary parts
of the hidden states. However, we found that such a nonlinearity usually performed poorly. Our intuition is that applying separate ReLU nonlinearities to the real and imaginary parts brutally impacts the phase of a complex number,
making it difﬁcult to learn structure.
We speculate that maintaining the phase of hidden states
may be important for storing information across a large
number of time steps, and our experiments supported this
claim. A variation of the ReLU that we name modReLU ,
is what we ﬁnally chose. It is a pointwise nonlinearity,
modReLU (z) :C!C, which affects only the absolute
value of a complex number, deﬁned as
modReLU (z) =(jzj+b)z
jzjifjzj+b0
0 ifjzj+b<0(8)
whereb2Ris a bias parameter of the nonlinearity. For a
nhdimensional hidden space we learn nhnonlinearity bias
parameters, one per dimension. Note that the modReLU
is similar to the ReLU in spirit, in fact more concretely
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio