Various architectural advancements in the design of recurrent neural networks (RNN) have been focusing on improving the empirical stability and representability by sacrificing the complexity of the architecture. However, more remains to be done to fully understand the fundamental trade-off between these conflicting requirements. Towards answering this question, we forsake the purely bottom-up approach of data-driven machine learning to understand, instead, the physical origin and dynamical properties of existing RNN architectures. This facilitates designing new RNNs with smaller complexity overhead and provable stability guarantee. First, we define a family of deep recurrent neural networks, n-t-ORNN, according to the order of nonlinearity n and the range of temporal memory scale t in their underlying dynamics embodied in the form of discretized ordinary differential equations. We show that most of the existing proposals of RNN architectures belong to different orders of n-t-ORNNs. We then propose a new RNN ansatz, namely the Quantum-inspired Universal computing Neural Network (QUNN), to leverage the reversibility, stability, and universality of quantum computation for stable and universal RNN. QUNN provides a complexity reduction in the number of training parameters from being polynomial in both data and correlation time to only linear in correlation time. Compared to Long Short Term Memory (LSTM), QUNN of the same number of hidden layers facilitates higher nonlinearity and longer memory span with provable stability. Our work opens new directions in designing minimal RNNs based on additional knowledge about the dynamical nature of both the data and different training architectures.
This paper attempts to do three things:
	1) introduce a generalization / formalism for describing RNN architectures
	2) demonstrate how various popular RNN architectures fit within the proposed framework
	3) propose a new RNN architecture within the proposed framework that overcomes some limitations in LSTMs
The ultimate goal of this work is to develop an architecture that:
	1) is better able to model long-term dependencies
	2) is stable and efficient to train
Some strengths, concerns, and questions loosely ordered by section:
Stable RNNs
	- it's not clear to me where equations (2) and (3) come from; what is the motivation? Is it somehow derived from this Runge-Kutta method (I'm not familiar with it)? 
	- I don't understand what this t^th order time-derivative amounts to in practice. A major claim (in Table 4) is that LSTMs are time-order 2 whereas QUNNs are time-order L and the implication is that this means LSTMs are worse at modeling long term structure than QUNNs ; but how does that actually relate to practical ability to model long-term dependencies? It certainly doesn't seem correct to me to say that LSTMs can only memorize sequences of length 2, so I don't know why we should care about this time-derivative order.
	- I thought this section was poorly written. The notation was poorly chosen at times, e.g. the t_k notation and the fact that some $l$ have subscripts and some don't. There were also some severe typos, e.g. I think Claim 1 should be "L-2-ORNN". Furthermore, there were crucially omitted definitions: what is reversibility and why should we care? Relatedly, the "proofs" are extremely hand-wavy and just cite unexplained methods with no further information.
QUNNs
	- The practical difference between QUNNs and LSTMs seems to be that the weights of the QUNN are dynamic within a single forward prop of the network, whereas LSTM weights are fixed given a single run (although the author does admit that the forget gates adds some element of dynamism, but there's no concrete evidence to draw conclusions about differences in practice).
	- I don't understand the repeated claim that LSTMs don't depend on the data; aren't the weights learned from data?
	
There may be something interesting in this paper, but it's not clear to me in its current incarnation and I'm not convinced that an eight-page conference paper is the right venue for such a work. There's a substantial of amount of exposition that needs to be there and is currently missing. I suspect the author knows this, but due to space constraints had to omit a lot of definitions and explanations.
I don't think all papers need experiments, but this paper I think would have greatly benefited from one. Community knowledge of LSTMs has reached a point where they are in practice easy to train and fairly stable (though admittedly with a lot of tricks). It would have been much more convincing to have simple examples where LSTMs fail due to instability and QUNNs succeed. Similarly, regarding long-term dependencies, my sense is that LSTMs are able to model some long-term dependencies. Experimental evidence of the gains offered by QUNNs would have also been very convincing.
Note: It looks like there's some funkiness in the tables on page 8 to fit into the page limit.