Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood. In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations. First, we show that simGD, as is, converges with stochastic sub-gradients under strict convexity in the primal variable. Second, we generalize optimistic simGD to accommodate an optimism rate separate from the learning rate and show its convergence with full gradients. Finally, we present anchored simGD, a new method, and show convergence with stochastic subgradients.
*Summary*
This paper provides the analysis of three algorithms in the context of minmax convex concave games: Simultaneous stochastic subgradient method, Simultaneous gradient with optimism and Simultaneous gradient with anchoring. These three algorithm are first analysed in continuous time with an ODE perspective and then leverage these intuitions and techniques to analyze the discrete time versions.
I think that these contributions are of interest of ICLR community, however, I have some concerns regarding the presentations of the results. 
*Decision*
I vote for a weak accept that could move to an accept if the authors improve the presentation of the paper:
The paragraph “Regularized dynamics and convergence’ in Section 3.1 in quite hard to follow. This subsection is basically the proof of the convergence of the continuous version of  GD-O. Stating the result before the proof would help the reader to understand where the authors want to go. 
Very same point for The subsection 4.1
For the stochastic version of SimGD-A a new parameter $\epsilon$ is introduced without any comment or description on why it is necessary. 
A FID above 40 for MNIST is very far from standard results (that are below 1). Thus I am not convinced by the practical advantage of Anchored Adam on these models that have performances results very far from the standard ones.  (for instance between 20 and 25 for CIFAR10 is very reasonable). It is maybe because you do not use convolutional layers in your architecture. I think that using a DGAN architecture (for instance the one from the pytorch tutorial https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html) would give the expected results (i.e. FID close to 0).
*Questions*
- When you cite Lassale Principle you mention that if $z_\infty$ is a limit point of $z(t)$ then starting at $z_\infty$, you stay at a constant distance to $z_*$. But in the bilinear example you give any point in the circle is not a limit point (size no dynamics converge to it). I guess when you said limit point you meant adherent point ?
- Is the condition $\epsilon$ only necessary for the proof ? Does $\epsilon =1$ work in practice ? 
If no, what is the best value for $\epsilon$ ?