STABLE RECURRENT MODELS
Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks. In this work, we conduct a thorough investigation of stable recurrent models. Theoretically, we prove stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent. Empirically, we demonstrate stable recurrent models often perform as well as their unstable counterparts on benchmark sequence tasks. Taken together, these findings shed light on the effective power of recurrent networks and suggest much of sequence learning happens, or can be made to happen, in the stable regime. Moreover, our results help to explain why in many cases practitioners succeed in replacing recurrent models by feed-forward models.
This is an interesting paper that I expect will generate some interest within the ICLR community and from deep learning researchers in general. The definition of stability is both intuitive and sound and the connection to exploding gradients is perhaps the most interesting and useful part of the paper. The sufficient conditions yield practical techniques for increasing the stability of, e.g., an LSTM, by constraining the weight matrices. They also show that stable recurrent models can be approximated by models with finite historical windows, e.g., truncated RNNs. Experiments in Sec 4 suggest that stable models produced by constraining standard RNN architectures can compete with their unconstrained unstable counterparts, and often without necessitating significant changes to architecture or hyperparameters. The perhaps most interesting observations are in Sec 4.3, in which the authors claim that even fundamentally unstable models, e.g., unconstrained RNNs, often operate in a stable regime, at least when being applied to in-sample data. I lean toward acceptance at the moment, but I am eager to discuss with the authors and other reviewers as I am not 100% confident that I fully understood the theory.
SUMMARY
This paper proposes a simple, generic definition of “stability” for recurrent, non-linear dynamical systems such as RNNs: that given two hidden states h, h’, the difference between their updated states given input x is bounded by the product between the difference between the states themselves and a small multiplier. The paper then immediately draws a connection between stability, asserting that unstable models are prone to gradient explosions during gradient descent-based training. In Sec 2.2, the paper presents sufficient conditions for basic RNNs and LSTMs to be stable. Secs 3.2 and 3.3 argue that stable recurrent models can be approximated by feedforward models during both inference and training with a finite history horizon, such as a RNN with a truncated history. Experiments in language and music modeling substantiate this claim: constrained, stable models are competitive with standard unconstrained models. Sec 4.3 sheds some light on this phenomenon, arguing that there is a weaker form of data-dependent stability and that even unstable models may operate in a stable regime for some problems, thus explaining the parity between stable and unstable models.
STRENGTHS
* This paper is surprisingly engaging and easy to read.
* The theorems are clearly stated and the proofs appear sound to me, though I will admit that I am not confident that I would catch a significant bug.
* This paper provides a new (to me, anyway) and thought-provoking analysis of RNNs. In particular, I was especially interested in the observation that stable models can be approximated by truncated models and that there is a connection between stability and long-term dependencies. This seems consistent with the fact that for many problems, non-recurrent models (ConvNets, Transformers, etc.) are often competitive with more complex architectures.
WEAKNESSES
* In practice it seems as though stability may depend on not only choice of  model architecture but also the data themselves. There is probably no good way to know a priori what the stability characteristics of a given data set are, making it tough to apply the ideas of this paper in practice
* The literature review seems a bit limited and appears to ignore the growing body of work on constraining RNN weight matrices to address both exploding and vanishing gradients. For example, I am pretty confident that the singular thresholding trick for renormalizing neural net weights has been  described in the literature previously.
* Although stable and unstable models appear to be competitive in experiments, the theoretical analysis provides no insights into stability and how it relates to accuracy.