Many tasks in modern machine learning can be formulated as finding equilibria in sequential games. In particular, two-player zero-sum sequential games, also known as minimax optimization, have received growing interest. It is tempting to apply gradient descent to solve minimax optimization given its popularity and success in supervised learning. However, it has been noted that naive application of gradient descent fails to find some local minimax and can converge to nonlocal-minimax points. In this paper, we propose Follow-the-Ridge (FR), a novel algorithm that provably converges to and only converges to local minimax. We show theoretically that the algorithm addresses the notorious rotational behaviour of gradient dynamics, and is compatible with preconditioning and positive momentum. Empirically, FR solves toy minimax problems and improves the convergence of GAN training compared to the recent minimax optimization algorithms. 2 +4xy−y 2 , our algorithm moves closer to the ridge every iteration and it moves along the ridge once it hits the ridge. Without the FR correction term, gradient dynamics can drift away from the ridge.
Summary: This paper designs a set of dynamics for learning in games called follow-the-ridge with the goal of finding local stackelberg equilibria. The main theoretical results show that the only stable attractors of the dynamics are stackelberg equilibria. Moreover, the authors give a deterministic convergence rate for the vanilla algorithm and a convergence rate using momentum. Empirical results show the learning dynamics cancel out rotational components and drive the vector field to zero rapidly, while reaching good performance on simple GAN examples.
Review: This paper focus on sequential games, which is the common formulation of GANs and a number of games in machine learning applications. From this perspective, it is natural to look at Stackelberg equilibria. In my opinion, the objective of the paper is important and relevant. The theoretical and empirical results are reasonably convincing. However, I do have some rather serious concerns about the general-sum game results and several questions regarding the relation to related work and the experiment details that need to be addressed.
1. The FR dynamics in algorithm 1 are closely related to the dynamics in [1]. In particular, the Jacobian of the FR dynamics is a similarity transform of the Jacobian of the dynamics in [1]. As a result, each algorithm has the same set of stable attractors. This should probably be mentioned in the paper. Given this relation, it is not clear what the advantage of the FR dynamics are over the dynamics in [1]. Could you please discuss this?
2. The gradient penalty regularization connection does not make sense in section 4.1. The optimization problem presented has an issue because the dimensions do not align in the constraint. The quantity \nabla_x f(x, y)^T H_yy^{-1}\nabla_x f(x, y) would not be defined if the dimensions of the players are not equal. 
3. In the related work it is claimed that two time-scale GDA converges only to local minimax and [2] is cited. I would avoid using this claim with respect to that paper since the statement following the main result in the paper is not right (see proposition 11 of [3] for proof). It is not clear what is meant when it is claimed that [1] can converge to non-local Stackelberg points. The dynamics in [1] only converge to local minimax points in the special case of zero-sum games.
4. Since the dynamics in the paper are the closest to those in [1], it seems that the paper would be stronger by comparing with that set of dynamics. 
5. I found it to be quite impressive that the vector field is driven to zero in the GAN examples. Just to clear, for each algorithm when the ‘gradient norm’ is shown, does this mean the norm of the update for each norm or does it mean the individual derivative for each player. For example in FR, would it be the norm of the derivative with respect to the follower variable of the function or the norm of the update including the second order information?
6. The path angle plot was interesting to see for the GAN example. The authors claim that the eigenvalues of the second order equilibria condition are non-negative. It would be nice if the authors could show the eigenvalues in the appendix and discuss how they were computed since it may be non-trivial to compute depending on the network size.
7. The damping method to stabilize training is not quite clear. Could you provide more details about how this was done?
My primary concerns have to do with the portion of the paper considering general-sum games. I do not understand where proposition 7 and 8 come from. I am not convinced the definitions provided are necessary and sufficient conditions for Stackelberg equilibria. In [1], a differential Stackelberg equilibrium is defined. The definition in this paper does not appear to agree with the definition in [1]. The final positive definite condition in proposition 7 and 8 does not appear to be taking the total derivative 2 times when I evaluate the derivatives, so I am not sure what the quantity is. If this is not a proper set of conditions for the equilibria, then it would also mean that the dynamics do not only converge to equilibria in general-sum games. It is important that the authors clear up this concern since I do not believe Theorem 3 holds as a consequence of problems with propositions 7 and 8.
[1] Fiez et al.,  "Convergence of Learning Dynamics in Stackelberg Games",  2019.
[2] Heusel et al., "GANs trained by a two time-scale update rule converge to a local Nash equilibrium", 2017.
[3] Mazumdar et al.,  "On Finding Local Nash Equilibria (and only Local Nash Equilibria) in Zero-Sum Games", 2019.
Post Author Response: Thanks to the authors for the effort in discussing the paper with me. The authors made several changes to the paper in response to my comments including removing section 4.1, fixing comments about related work, including details on the damping procedure, showing experimental comparisons to [1] along with an explanation of why the dynamics in this paper may be preferred for training GANs, providing details on propositions 7 and 8 and including reference to [1], adding further assumptions on the functions, and attempting to make theorem 3 more clear. Overall, I think this paper proposes an interesting set of dynamics, several meaningful theoretical guarantees, and impressive empirical results. I would be curious to see how it performs on even more large-scale GAN problems in the future. As a result, I have changed my original score from a weak reject to a weak accept. My primary concerns with the paper regarded the general-sum convergence results and I appreciated the explanations from the authors. I am still of the opinion that theorem 3 could be stated more rigorously in the sense that the neighborhood on which the local convergence holds should be more explicit. It seems to me that the convergence result may only hold in a ball around an equilibrium in which the implicit function is well-defined and the FR dynamics will be attracted to r(x) and that this space could be arbitrarily small for some problems. Nonetheless, this result is only in the appendix, and the paper includes enough contributions beyond this to warrant acceptance.