A VARIATIONAL INEQUALITY PERSPECTIVE ON GENERATIVE ADVERSARIAL NETWORKS
Generative adversarial networks (GANs) form a generative modeling approach known for producing appealing samples, but they are notably difficult to train. One common way to tackle this issue has been to propose new formulations of the GAN objective. Yet, surprisingly few studies have looked at optimization methods designed for this adversarial training. In this work, we cast GAN optimization problems in the general variational inequality framework. Tapping into the mathematical programming literature, we counter some common misconceptions about the difficulties of saddle point optimization and propose to extend techniques designed for variational inequalities to the training of GANs. We apply averaging, extrapolation and a computationally cheaper variant that we call extrapolation from the past to the stochastic gradient method (SGD) and Adam.
This paper looks at solving optimization problems that arise in GANs, via a variational inequality perspective (VIP). VIP entails solving an optimization problem that is related to the first order condition of the optimization problem that we wish to solve. VIP have been very successful in solving min-max style problems. Given that, GAN formulations tend to be min-max style problems (though not necessarily 0 sum) the VIP perspective is very natural, though under-explored in machine learning. Two techniques that have been widely used to solve VIP problems are averaging and extragradient methods. The authors look at a simple GAN setup where both the generator and the discriminator are linear models. In this case two kinds of gradient updates can be derived. First are simultaneous updates, and the other is alternated updates. The authors show that simultaneous updates are not even bounded and diverge to infinity, whereas alternated updates are more stable and stay bounded, but need not necessarily converge. However, I think this behaviour is limited to only linear discriminator/generator and might not extend beyond the linear case. The second key idea is the use of extra-gradient updates. Extra-gradient updates perform an "extra" or fake gradient step to get to a new point, and then kind of retracks back and perform a gradient step using the gradient step obtained from the "extra step".  This extra-gradient method is a close approximation to Euler's method, though far more computationally efficient.  However, the extragradient step requires one to calculate gradient twice, which can be expensive in large models. For this reason, the authors suggest using gradients from past as the "extragradient" in the extragradient method. 
For strongly-monotone operators (a generalization of strongly-convex functions) extrapolation updates are shown to have linear convergence.  Furthermore, the authors show that using extrapolation and averaging under the assumption that the operator is monotonic, and using constant step size SGD the rates of convergence are better than the rates obtained using plain SGD with averaging but without extrapolation. Authors also show how one can use these ideas using other first order methods such as ADAM instead of SGD. Experiments are shown on the DCGAN architecture. 
On the whole this is a really nice paper, that shows how standard ideas from VIP can be useful for training GANs. I recommend acceptance