Gated recurrent units (GRUs) were inspired by the common gated recurrent unit, long short-term memory (LSTM), as a means of capturing temporal structure with less complex memory unit architecture. Despite their incredible success in tasks such as natural and artificial language processing, speech, video, and polyphonic music, very little is understood about the specific dynamic features representable in a GRU network. As a result, it is difficult to know a priori how successful a GRU-RNN will perform on a given data set. In this paper, we develop a new theoretical framework to analyze one and two dimensional GRUs as a continuous dynamical system, and classify the dynamical features obtainable with such system. We found rich repertoire that includes stable limit cycles over time (nonlinear oscillations), multi-stable state transitions with various topologies, and homoclinic orbits. In addition, we show that any finite dimensional GRU cannot precisely replicate the dynamics of a ring attractor, or more generally, any continuous attractor, and is limited to finitely many isolated fixed points in theory. These findings were then experimentally verified in two dimensions by means of time series prediction.
The authors analyse GRUs with hidden sizes of one and two as continuous-time dynamical systems, claiming that the expressive power of the hidden state representation can provide prior knowledge on how well a GRU will perform on a given dataset. Their analysis shows what kind of hidden state dynamics the GRU can approximate in one and two dimensions. In the experimental part, they show how a GRU with two hidden states trained on a multistep prediction task can learn such dynamics.
Although RNNs are important for Machine Learning, the paper seems to contain flaws in the theoretical part, which seem to invalidate some of the claimed results. But we may change our rating in case of a convincing rebuttal.
The Proof of Lemma 2 claims that h(t) achieves all values on the real set, which is false (h(t) assumes values in (-1,1)). Nevertheless, the theorem should hold since there is always at least one intersection between h and tanh(f(h)).
Lemma 1 claims that for any choice of parameters, there exist only finitely many fixed points. However, in the proof the authors only show that the number of fixed points cannot be uncountable, without taking into consideration the possibility that there are countably many fixed points. The proof also omits steps concerning the Taylor expansion which would make the proof clearer: We suggest adding those steps in the appendix. Furthermore, when equation (12) is Taylor-expanded, the authors do not consider the case where the GRU parameters are such that the argument of function “sech” is outside its convergence radius. These might be parameters for which there are infinitely many fixed points, even if we are unable to provide a Taylor expansion. The Lemma may still be correct, but this does not seem to be a complete proof.
The authors claim that an arbitrarily close approximation of a line attractor can be created using two GRUs, but no proof is provided.
The experimental part is difficult to evaluate since there are no learning curves for the three tasks. For instance, it is difficult to judge whether the GRUs are unable to learn the dynamics of a ring attractor because of theoretical limitations or because the model has not been properly trained for the specific task.
The paper is easy to read, except for certain parts where it is not clear if some of the statements are true in general or just have not been proven false by the authors. It is not clear why Figure 3 is representing all possible simple fixed points and bifurcation fixed points: is there a theoretical result stating that these are the only possible topologies, or are these the only ones found? The same question applies for the 36 images in figure 9. The range of the parameters used for finding these configurations is not specified.
Since the hidden state assumes values in (-1,1)^2, why is its range in most of the images (-1.5,1.5)?
We are not familiar with related work on transformations from discrete to continuous dynamical systems: are the dynamics of the discrete time GRU model preserved in the transformation? If so, is there a reference for this? Are the phase portraits in the middle row of figure 8 generated by letting the discrete GRU system evolve, or is the continuous system used with the parameters of the trained GRU?
We would like to see more explanations of why various topologies are useful for the applications mentioned in the paper. Given a generic dataset, how can these results help to understand how well a GRU will perform?
What is the reason behind the belief that the analysis extends to higher dimensions? The effects of a 1D -> 2D extension are far from trivial - why should that be different for higher dimensions?
The problem the authors want to solve seems important, and some of the theoretical results are promising, but we think that this paper has to be further polished before acceptance.
It is possible that we will increase the score if the authors can provide clarifications on the above questions.
Additional comments:
Introduction
-The vanishing gradient problem was not discovered in 1994, but in 1991 by Hochreiter: 
Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TU Munich, 1991. Advisor J. Schmidhuber.
- Make clear that GRU is a variant of vanilla LSTM with forget gates (where one gate is missing):
Gers et al. “Learning to Forget: Continual Prediction with LSTM.“ Neural Computation, 12(10):2451-2471, 2000. 
- The intro says that GRU has become widely popular and cites Britz et al 2017, but Britz et al actually show that LSTM consistently outperforms its variant GRU in Neural Machine Translation. Please clarify this. 
- Also mention Weiss et al (“On the Practical Computational Power of Finite Precision RNNs for Language Recognition”) who exhibited basic limitations of GRU when compared to LSTM. 
- Is the result by Weiss et al actually related to the result of the authors who found that 2 GRUS cannot accurately a line attractor without near zero constant curvature in the phase space?
Section 2
-Wrong brackets in equation (4)
-Missing bracket before citing Laurent & Brecht
Section 4
-”We conjecture that the system depicted in figure 2A..” Should be figure 3A
- Lemma1: UZ has capital Z subscript
Section 5.2
-”The added affine transformation allows for a sufficiently long subinterval”: “sufficiently long” is too vague
Section 5.3
“A manifold with without near zero constant curvature”: should be “a manifold without near zero constant curvature”
Appendix A
-Wrong brackets in equation (20)
Appendix B
- In the proof of Theorem 1, the derivative is of (29), not of (12)
Appendix C
Figure 9: “who’s initial conditions” should be “whose initial conditions”
After rebuttal:
It's better now. However, the revised introduction still says:  "GRU has become wildly popular in the machine learning community thanks to its performance in machine translation (Britz et al., 2017) ... LSTM has been shown to outperform GRU on neural machine translation (Britz et al., 2017).... specifically unbounded counting, come easy to LSTM networks but not to GRU networks (Weiss et al., 2018)." 
So better remove the first statement on Britz et al: "GRU has become wildly popular ... in machine translation (Britz et al., 2017)" because they actually show why GRU is NOT wildly popular in machine translation, as correctly justified later in the same paragraph.
Pending the above revision, we'd like to increase our evaluation by 2 points, up to 6!