In this paper, we propose a mix-generator generative adversarial networks (PGAN) model that works in parallel by mixing multiple disjoint generators to approximate a complex real distribution. In our model, we propose an adjustment component that collects all the generated data points from the generators, learns the boundary between each pair of generators, and provides error to separate the support of each of the generated distributions. To overcome the instability in a multiplayer game, a shrinkage adjustment component method is introduced to gradually reduce the boundary between generators during the training procedure. To address the linearly growing training time problem in a multiple generators model, we propose a method to train the generators in parallel. This means that our work can be scaled up to large parallel computation frameworks. We present an efficient loss function for the discriminator, an effective adjustment component, and a suitable generator. We also show how to introduce the decay factor to stabilize the training procedure. We have performed extensive experiments on synthetic datasets, MNIST, and CIFAR-10. These experiments reveal that the error provided by the adjustment component could successfully separate the generated distributions and each of the generators can stably learn a part of the real distribution even if only a few modes are contained in the real distribution.
Overall, the writing is very confusing at points and needs some attention to make the paper clearer. I’m not entirely sure the authors understand the material particularly well, as I found some of the arguments and narrative confusing or just incorrect. I don’t really see any significant contribution here except “we had this idea for this model, and it works”. There’s no interesting questions being asked about missing modes (and no answers through good experimentation), no insight that might contribute to our understanding of the problem, and no comparison to other models. My guess is this submission was rushed (and perhaps they were just looking for feedback). I like the idea, don’t get me wrong: a model that is trainable across multiple GPUs and that distributes generative work is pretty cool, and I want to see this work succeed (after a *lot* more work). But the paper really lacks what I’d consider good science, and I don’t see it publishable without significant improvement.
Personally I think you should change the angle from missing modes to parallel training. I don’t see any strong guarantees that the model will do what you say it will, especially as beta goes to zero.
Detailed comments
P1
“, that explicitly approximate data distribution, the approximation of GAN is implicit”
The wording of this is pretty strange: by “implicit”, we mean that we only have *samples* from the distribution(s) of interest, but what does it mean for an approximation to be “implicit”?
From the intro, it doesn’t sound like the approach is meant for the “mode collapse” problem, but for dealing with missing modes. These are different types of failures for GANs, and while there are many theories for why these happen, to my knowledge there’s no such consensus that these issues are the same.
For instance, what is keeping each of the generators from collapsing onto a single value? We often see the model collapse on several different values: why couldn’t each of your generators do this?
P2: No, it is incorrect that the KL is what is causing mode collapse, and I think actually you mean “missing modes”. Arjovsky et al addresses the mode collapse problem, which is just another word for a type of instability in GANs. But this isn’t because of “vanishing gradients”, as the “proxy loss” (which you call “heuristic loss”, this isn’t a common term, fyi), which is what GANs are trained on in practice don’t vanish, but show some other sorts of instabilities (Arjovsky 2016). That said, other GAN variants without regularization also show collapse *and* missing modes, such as LSGAN and all the f-GAN variants (even the auto encoder variants).
You should also probably cite Che et al 2016 as another model that addressed missing modes. Also, what about ALI, BiGAN, and ALiCE? These also address missing modes (at least they claim to).
I don’t understand why you’re comparing f-GAN and WGAN convergences: they are addressing different things with GANs: one shows insight into what exactly traditional GANs are doing (solving a dual problem of minimizing an f-divergence) versus addressing stability through using an IPM (though also a dual formulation of the wasserstein). f-GANs ensure neither stability nor non-vanishing gradients.
P3: I like the breakdown of how the memory is organized.
This is for multi-GPU, correct? This needs to be explicitly stated.
P6:
There’s a sign error in proof 1 (both in the definition of the reverse KL and when the loss is written out).
Also, the gradient w.r.t. theta magically appears in the second half.
This is a pretty round-about way to arrive at that you’re minimizing the reverse KL: I’m pretty sure this can be shown by formulating the second term in f-gan (the one where you sample from the generator), that is f*(T), where f* is the convex conjugate of f = -log
Mixture of Gaussians: common *missing modes* experiment.
So my general comments about the experiments
You need to compare to other models that address missing modes. Overall, many people have shown success with experiments similar to your simple mixture of Gaussians experiments, so in order to show something significant here, you will need to have a more challenging experiments and show a comparison to other models.
The real-world experiments are fairly unconvincing, as you only show MNIST and CIFAR-10 (and MNIST doesn’t look very good). Overall, the good inception scores aren’t too surprising given the model has several generators for each mode, but I think we need to see a demonstration on better datasets.