Generative Adversarial Networks (GANs) have been proposed as an approach to learning generative models. While GANs have demonstrated promising performance on multiple vision tasks, their learning dynamics are not yet well understood, neither in theory nor in practice. In particular, the work in this domain has been focused so far only on understanding the properties of the stationary solutions that this dynamics might converge to, and of the behavior of that dynamics in this solutions' immediate neighborhood. To address this issue, in this work we take a first step towards a principled study of the GAN dynamics itself. To this end, we propose a model that, on one hand, exhibits several of the common problematic convergence behaviors (e.g., vanishing gradient, mode collapse, diverging or oscillatory behavior), but on the other hand, is sufficiently simple to enable rigorous convergence analysis. This methodology enables us to exhibit an interesting phenomena: a GAN with an optimal discriminator provably converges, while guiding the GAN training using only a first order approximation of the discriminator leads to unstable GAN dynamics and mode collapse. This suggests that such usage of the first order approximation of the discriminator, which is a de-facto standard in all the existing GAN dynamics, might be one of the factors that makes GAN training so challenging in practice. Additionally, our convergence result constitutes the first rigorous analysis of a dynamics of a concrete parametric GAN.
The authors proposes to study the impact of GANS in two different settings:
1. at each iteration, train the discriminator to convergence and do a (or a few) gradient steps for updating the generator
2. just do a few gradient steps for the discriminator and the generator
This is done in a very toy example: a one dimensional equally weighted mixture of two Gaussian distributions.
Clarity: the text is reasonably well written, but with some redundancy (e.g. see section 2.1) , and quite a few grammatical and mathematical typos here and there. (e.g. Lemma 4.2., $f$ should be $g$, p7 Rect(0) is actually the empty set, etc..)
Gaining insights into the mechanics of training GANs is indeed important. The authors main finding is that, in this very particular setting, it seems that training the discriminator to convergence leads to convergence. Indeed, in real settings, people have tried such strategies for WGAN for examples. For standard GANs, if one adds a little bit of noise to the labels for example, people have also reported good result for such a strategy (although, without label smoothing, this will indeed leads to problems).
Although I have not checked all the mathematical fine details, the approach/proof looks sound (although it is not at all clear too me why the choice of gradient step-sizes does not play a more important roles the the stated results). My biggest complain is that the situation analyzed is so simple (although the convergence proof is far from trivial) that I am not at all convinced that this sheds much light on more realistic examples. Since this is the main meat of the paper (i.e. no methodological innovations), I feel that this is too little an innovation for deserving publication in ICLR2018.