  We show that both an LSTM and a unitary-evolution recurrent neural network
(URN) can achieve encouraging accuracy on two types of syntactic patterns:
context-free long distance agreement, and mildly context-sensitive cross serial
dependencies. This work extends recent experiments on deeply nested
context-free long distance dependencies, with similar results. URNs differ from
LSTMs in that they avoid non-linear activation functions, and they apply matrix
multiplication to word embeddings encoded as unitary matrices. This permits
them to retain all information in the processing of an input string over
arbitrary distances. It also causes them to satisfy strict compositionality.
URNs constitute a significant advance in the search for explainable models in
deep learning applied to NLP.

Computation and Language