THE UNUSUAL EFFECTIVENESS OF AVERAGING IN GAN TRAINING
We examine two different techniques for parameter averaging in GAN training. Moving Average (MA) computes the time-average of parameters, whereas Exponential Moving Average (EMA) computes an exponentially discounted sum. Whilst MA is known to lead to convergence in bilinear settings, we provide theto our knowledge -first theoretical arguments in support of EMA. We show that EMA converges to limit cycles around the equilibrium with vanishing amplitude as the discount parameter approaches one for simple bilinear games and also enhances the stability of general GAN training. We establish experimentally that both techniques are strikingly effective in the non-convex-concave GAN setting as well. Both improve inception and FID scores on different architectures and for different GAN objectives. We provide comprehensive experimental results across a range of datasets -mixture of Gaussians, CIFAR-10, STL-10, CelebA and ImageNet -to demonstrate its effectiveness. We achieve state-of-the-art results on CIFAR-10 and produce clean CelebA face images.
This paper tries to adapt the concept of averaging, well known is the game literature, to GAN training. In a simple min-max example the iterates obtained by gradient method do not converge to the equilibrium of the game but their average does. This work first provides intuitions on the potential benefits of exponential moving average (EMA) on a simple illustrative example and explore the effect of averaging on GAN.
In think that the approach of this paper is interesting. I particularly like the experiments on Celeb-A (Fig 6 and 7) that seem to show that the averaged iterates change more smoothly (with respect to the attributes of the faces) during the training procedure. Nevertheless, I have some concerns about the claims of the paper and the experimental process.
I'm surprised by the values of the inception score provided in Table 2 which do not seem to correlate with the sample quality in Fig. 3. Why did not you use the standard implementation of the inception score provided in Salimans et al. [2016]'s paper ?
I think that the effectiveness of EMA over uniform averaging is a bit overclaimed. 
- From a theoretical point of view uniform averaging works better (at least in your example in 3.1): If you (uniformly) average the periodic orbit you get a converging iterate. Moreover, concerning to this toy example, note that this continuous analysis has been already introduced in [Goodfellow et al., 2016] and the Hamiltonian interpretation has been already provided in [Balduzzi et al. 2018].
However I think that the intuition on the vanishing magnitude of the oscillation provided by EMA is interesting.
- The continuous dynamics is actually different from the discrete one, I think that an analysis on the discrete case that is used in practice might be more insightful.
- The comparison with uniform averaging is not fair in the sense that uniform averaging has no hyperparameter to tune: In figure 6 uniform averaging performs better than a not well tuned EMA. A fair comparison would be for instance to propose a parametrized online averaging $\theta_{MA}^t = \frac{t - \alpha}{t} \theta_{MA}^{t-1} + \frac{\alpha}{t} \theta_t$ and to tune it the same way $\beta$ is tuned in EMA.
Refs:
Salimans, Tim, et al. "Improved techniques for training gans." Advances in Neural Information Processing Systems. 2016.
Goodfellow, I. (2016). NIPS 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160.
Balduzzi, David, et al. "The Mechanics of n-Player Differentiable Games." ICML (2018).
Minor comments: 
- In the introduction "gradient vector fields of the game may not be conservative (Mescheder et al. 2017)" and the related work "Mescheder et al. (2017) states that a reason for non-convergence is the non-conservative gradient
vector of the players.": the notion of conservative vs. non-conservative vector field is never mentioned in [Mescheder et al. 2017]. I think you are actually referring to the blog post on that paper https://www.inference.vc/my-notes-on-the-numerics-of-gans/ .
- In the Related work "can not"
- "In fact, it has recently been established that the smooth (continuous-time) analogues of first order methods such as online gradient descent (follow-the-regularized leader) in bilinear zero-sum games are recurrent (i.e. effectively periodic)
with trajectories cycling back into themselves. " can you provide a citation ?
- Some published papers are refereed as arxiv paper ( for instance (Mescheder et al. 2017) and (Mescheder et al. 2018)), you should cite the published version.