ANTISYMMETRICRNN: A DYNAMICAL SYSTEM VIEW ON RECURRENT NEURAL NETWORKS
Recurrent neural networks have gained widespread use in modeling sequential data. Learning long-term dependencies using these models remains difficult though, due to exploding or vanishing gradients. In this paper, we draw connections between recurrent networks and ordinary differential equations. A special form of recurrent networks called the AntisymmetricRNN is proposed under this theoretical framework, which is able to capture long-term dependencies thanks to the stability property of its underlying differential equation. Existing approaches to improving RNN trainability often incur significant computation overhead. In comparison, AntisymmetricRNN achieves the same goal by design. We showcase the advantage of this new architecture through extensive simulations and experiments. AntisymmetricRNN exhibits much more predictable dynamics. It outperforms regular LSTM models on tasks requiring long-term memory and matches the performance on tasks where short-term dependencies dominate despite being much simpler.
In this paper, the authors provide a new approach to analyze the behavior of
RNNs by relating the RNNs with ODE numerical schemes. They provide analysis on
the stability of forward Euler scheme and proposed an RNN architecture called
AntisymmetricRNN to solve the gradient exploding/vanishing problem. 
The paper is well presented although more recent works in this direction
should be cited and discussed. Also, some important issues are omitted and not
explained. 
For example, the analysis begins with "RNNs with feedback" rather than vanilla
RNN, since vanilla RNN does not have the residual structure as eq(3). The
authors should note that clearly in the paper. 
Although there are previous works relating ResNets with ODEs, such as [1],
this paper is original as it is the first work that relates the stability of
ODE numerical scheme with the gradient vanishing/exploding issues in RNNs. 
In general, this paper provides a novel approach to analyze the gradient
vanishing/exploding issue in RNNs and provides applicable solutions, thus I
recommend to accept it. 
Detailed comments:
The gradient exploding/vanishing issue has been extensively studied these
years and more recent results should be discussed in related works.
Author mentioned that existing methods "come with significant computational
overhead and reportedly hinder representation power of these models". However
this is not true for [2] which achieves full expressive power with
no-overhead. 
It is true that "orthogonal weight matrices alone does not prevent exploding
and vanishing gradients", thus there are architectural approaches that can
bound the gradient norm by constants [3]. 
The authors argued that the critical criterion is important in preserving the
gradient norm. However, later on added a diffusion term to maintain the
stability of forward Euler method. Thus the gradient will vanish
exponentially w.r.t. time step t as: (1-\gamma)^t. Could the authors provide
more detailed analysis on this issue? 
Since eq(3) cannot be regarded as vanilla RNN, it would be better begin the
analysis with advanced RNN architectures that fit in this form, such as
Residual RNN, Statistical Recurrent Units and Fourier Recurrent Units. 
Why sharing the weight matrix of gated units and recurrent units? Is there any
other reason to do this other than reducing the number of parameters?
More experiment should be conducted on real applications of RNN, such as
language model or machine translation. 
[1] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer
neural networks: Bridging deep architectures and numerical differential
equations. In ICML, pp. 3276–3285, 2018.  
[2] Zhang, Jiong, Qi Lei, and Inderjit S. Dhillon. "Stabilizing Gradients for
Deep Neural Networks via Efficient SVD Parameterization." In ICML, pp.
5806-5814, 2018.
[3] Zhang, Jiong, Yibo Lin, Zhao Song, and Inderjit S. Dhillon. "Learning Long
Term Dependencies via Fourier Recurrent Units." In ICML, pp. 5815-5823, 2018.