Gradient descent GAN optimization is locally stable
Despite the growing prominence of generative adversarial networks (GANs), optimization in GANs is still a poorly understood topic. In this paper, we analyze the "gradient descent" form of GAN optimization, i.e., the natural setting where we simultaneously take small gradient steps in both generator and discriminator parameters. We show that even though GAN optimization does not correspond to a convex-concave game (even for simple parameterizations), under proper conditions, equilibrium points of this optimization procedure are still locally asymptotically stable for the traditional GAN formulation. On the other hand, we show that the recently proposed Wasserstein GAN can have non-convergent limit cycles near equilibrium. Motivated by this stability analysis, we propose an additional regularization term for gradient descent GAN updates, which is able to guarantee local stability for both the WGAN and the traditional GAN, and also shows practical promise in speeding up convergence and addressing mode collapse.
The authors present a dynamical system based analysis of simultaneous gradient descent updates for GANs, by considering the limit dynamical system that corresponds to the discrete updates. They show that under a series of assumptions, an equilibrium point of the dynamical system is locally asymptotically stable, implying convergence to the equilibrium if the system is initialized in a close neighborhood of it. Then they show how some types of GANs fail to satisfy some of their conditions and propose a fix to the gradient updates that re-instate local stability. They give experimental evidence that the local-stability inspired fix yields improvements in practice on MNIST digit generation and simple multi-modal distributions. 
However, I do think that these drawbacks are remedied by the fact that their modification, based on local asymptotic theory, did give noticeable improvements. Moreover, the local stability proof is a bit non-trivial from theoretical point of view.
I think this paper provides some nice theoretical results in GAN training and gives interesting insights to GAN training through the dynamical system lens. The most promising part of the paper is the fact that their theory inspired fix to gradient descent yields experimental improvements. 
The theory part of the paper has some drawbacks: the assumptions that under equilibrium the discriminator outputs always zero is fairly strong and hard to believe in practice. In particular, it's hard to believe that the generator will be outputting a completely indistinguishable distribution from the real data, which is the only case where this could be an equilibrium for the discriminator. Also the fact that this is only a local stability without any arguments about the size of the region of attraction, might make the result irrelevant in practical training, where initializing in a tiny ball around the equilibrium would be impossible. Finally, it is not clear that local stability of the limit dynamical system will also imply local stability of the discrete system with stochastic gradients: if the variance in the stochastic gradient is large and the region of attraction is tiny, then a tiny step outside of the equilibrium together with a high variance gradient step could push the system outside of the region of attraction.