Generative adversarial networks have been very successful in generative modeling, however they remain relatively challenging to train compared to standard deep neural networks. In this paper, we propose new visualization techniques for the optimization landscapes of GANs that enable us to study the game vector field resulting from the concatenation of the gradient of both players. Using these visualization techniques we try to bridge the gap between theory and practice by showing empirically that the training of GANs exhibits significant rotations around LSSP, similar to the one predicted by theory on toy examples. Moreover, we provide empirical evidence that GAN training seems to converge to a stable stationary point which is a saddle point for the generator loss, not a minimum, while still achieving excellent performance.
Summary: 
This paper proposes visualization techniques for the optimization landscape in GANs. The primary tool presented in this paper is a quantity called path-angle, which looks at the angle between the game vector field and the linear path between a point away from a stationary point and a point near a stationary point. The paper present examples of the visualization for dynamics with pure attraction, pure rotation, and a mix of attraction and rotation. Along with this, the authors propose to look at the eigenvalues of the game Jacobian and the individual player Hessian’s to evaluate convergence in GANs. The paper presents application of the tools on GANs trained with NSGAN and WGAN-GP objectives on a mixture of Gaussians, MNIST, and CIFAR10. The primary observation is that the generator performance is good, but the algorithms converge to non-Nash stable attractors. Moreover, it is shown using the path-angle plots that GANs exhibit rotational behavior around stable points.
Review: 
There has been a lot of work in the past few years (and ongoing) on principled training approaches for GANs. The objective of the algorithms is typically to converge a differential Nash equilibrium and/or to reach a stable point of the dynamics quickly. In my view, this work fills some of the gap on the empirical side of things with respect to each goal. 
Notably, a main idea to speed up convergence in GANs is to change the gradient play dynamics so rotational components are neutralized. The path angle visualization provides a novel tool to evaluate the empirical ability of any dynamics proposed for GANs to cancel out rotational components. Since it is generally known that gradient play dynamics are susceptible to cycling, I would have been interesting in seeing the path angle plots for some recently proposed algorithms such as consensus, symplectic gradient adjustment, stable opponent shaping, local symplectic surgery, etc to see how they compare. This would have made the experiments using the path angle visualization stronger in my view. Nonetheless, the path angle tool is useful and I can foresee it being commonly used in the future. 
Aside from neutralizing rotational components, dynamics have been proposed with the goal of avoiding non-Nash stable attractors and converging only to differential Nash equilibria. However, to my knowledge, there has not been much, if any, evaluation in GANs to see if the methods are in fact converging to Nash equilibria as theory may predict. While simple, I found it interesting to evaluate the eigenvalues of the relevant quantities at convergence. I am curious why the authors evaluate the top-k eigenvalues in terms of magnitude? The scipy package referenced in the appendix can compute the largest and smallest real eigenvalues, which is what it seems like you would want to evaluate the definiteness of the game Jacobian and the individual player Hessians. The most interesting empirical result in the paper to me was that it is common to converge to non-Nash stable attractors using standard training techniques and at such stable points the generator performance is strong. This is an important observation and  may cause some consideration of what points should be sought in GANs. I am not fully convinced this is always what the dynamics would always converge to depending on the network, learning rates, optimization methods, etc, but showing that it can be the case is useful. 
Overall, I think this paper introduces some useful tools to interpret the performance in GANs and to help understand the behavior of training dynamics. The main tool introduced was the path angle visualization and the primary empirical result was that standard GAN methods may reach non-Nash stable attractors and perform well. The paper probably be condensed in the first 4 pages, so that more experimental results could be presented and this would make the paper stronger.  
Post Response: Thanks for the response. I believe this paper should be accepted.