suffer from saturating hidden states as much as LSTMs,
typical RNNs, or RNNs initialized with the identity weight
matrix (IRNNs).
Unitary Evolution Recurrent Neural Networks
Acknowledgments : We thank the developers of
Theano (Bergstra et al., 2010) for their great work.
We thank NSERC, Compute Canada, Canada Research
Chairs and CIFAR for their support. We would also like
to thank C ¸ aglar Gulc ¸ehre, David Krueger, Soroush Mehri,
Marcin Moczulski, Mohammad Pezeshki and Saizheng
Zhang for helpful discussions, comments and code sharing.
References
Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo.
Learning long-term dependencies with gradient descent
is difﬁcult. IEE Transactions on Neural Networks , 5,
1994.
Bergstra, James, Breuleux, Olivier, Bastien, Fr ´ed´eric,
Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guillaume, Turian, Joseph, Warde-Farley, David, and Bengio, Yoshua. Theano: a CPU and GPU math expression
Recurrent neural networks (RNNs) are notoriously difficult to train. When the
eigenvalues of the hidden to hidden weight matrix deviate from absolute value
1, optimization becomes difficult due to the well studied issue of vanishing
and exploding gradients, especially when trying to learn long-term
dependencies. To circumvent this problem, we propose a new architecture that
learns a unitary weight matrix, with eigenvalues of absolute value exactly 1.
The challenge we address is that of parametrizing unitary matrices in a way
that does not require expensive computations (such as eigendecomposition) after
each weight update. We construct an expressive unitary weight matrix by
composing several structured matrices that act as building blocks with
parameters to be learned. Optimization with this parameterization becomes
feasible only when considering hidden states in the complex domain. We
demonstrate the potential of this architecture by achieving state of the art
results in several hard tasks involving very long-term dependencies.
Martin Arjovsky Amar Shah Yoshua Bengio