MGAN: TRAINING GENERATIVE ADVERSARIAL NETS WITH MULTIPLE GENERATORS
We propose in this paper a new approach to train the Generative Adversarial Nets (GANs) with a mixture of generators to overcome the mode collapsing problem. The main intuition is to employ multiple generators, instead of using a single one as in the original GAN. The idea is simple, yet proven to be extremely effective at covering diverse data modes, easily overcoming the mode collapsing problem and delivering state-of-the-art results. A minimax formulation was able to establish among a classifier, a discriminator, and a set of generators in a similar spirit with GAN. Generators create samples that are intended to come from the same distribution as the training data, whilst the discriminator determines whether samples are true data or generated by generators, and the classifier specifies which generator a sample comes from. The distinguishing feature is that internal samples are created from multiple generators, and then one of them will be randomly selected as final output similar to the mechanism of a probabilistic mixture model. We term our method Mixture Generative Adversarial Nets (MGAN). We develop theoretical analysis to prove that, at the equilibrium, the Jensen-Shannon divergence (JSD) between the mixture of generators' distributions and the empirical data distribution is minimal, whilst the JSD among generators' distributions is maximal, hence effectively avoiding the mode collapsing problem. By utilizing parameter sharing, our proposed model adds minimal computational cost to the standard GAN, and thus can also efficiently scale to large-scale datasets. We conduct extensive experiments on synthetic 2D data and natural image databases (CIFAR-10, STL-10 and ImageNet) to demonstrate the superior performance of our MGAN in achieving state-of-the-art Inception scores over latest baselines, generating diverse and appealing recognizable objects at different resolutions, and specializing in capturing different types of objects by the generators.
The present manuscript attempts to address the problem of mode collapse in GANs using a constrained mixture distribution for the generator, and an auxiliary classifier which predicts the source mixture component, plus a loss term which encourages diversity amongst components.
All told the proposed method is quite incremental, as mixture GANs/multi-generators have been done before. The Inception scores are good but it's widely known now that Inception scores are a deeply flawed measure, and presenting it as the only quantitative measure in a manuscript which makes strong claims about mode collapse unfortunately will not suffice. If the generator were to generate one template per class for which the Inception network's p(y|x) had low entropy, the Inception score would be quite high even though the model had only memorized one image per class. For claims surrounding mode collapse in particular, evaluation against a parameter count matched baseline using the AIS log likelihood estimation procedure in Wu et al (2017) would be the gold standard. Frechet Inception distance has also been proposed which at least has some favourable properties relative to Inception score.
The mixing proportions are fixed to the uniform distribution, and therefore this method also makes the unrealistic assumption that modes are equiprobable and require an equal amount of modeling capacity. This seems quite dubious.
Finally, their own qualitative results indicate that they've simply moved the problem, with clear evidence of mode collapse in one of their mixture components in figure 5c, 4th row from the bottom. Indeed, this does nothing to address the problem of mode collapse in general, as there is nothing preventing individual mixture component GANs from collapsing.
Uncited prior work includes Generative Adversarial Parallelization of Im et al (2016). Also, if I'm not mistaken this is quite similar to an AC-GAN, where the classes are instead randomly assigned and the generator conditioning is done in a certain way; namely the first layer activations are the sum of K embeddings which are gated by the active mixture component. More discussion of this would be warranted.
Other notes:
- The introduction contains no discussion of the ill-posedness of the GAN game as it is played in practice.
- "As a result, the optimization order in 1 can be reversed" this does not accurately characterize the source of the issues, see, e.g. Goodfellow (2015) "On distinguishability criteria...".
- Section 3: the second last sentence of the third paragraph is vague and doesn't really say anything. Of course parameter sharing leverages common information. How does this help to train the model effectively?
- Section 3: Since JSD is defined between two distributions, it is not clear what JSD_pi(P_G1, P_G2, ...) refers to. The last line of the proof of theorem 2 leaps to calling this term a Jensen-Shannon divergence but it's not clear what the steps are; it looks like a regular KL divergence to me.
- Section 3: Also, is the classifier being trained to maximize this divergence or just the generator? I assume the latter.
- The proof of Theorem 3 makes unrealistic assumptions that we know the number of components a priori as well as their mixing proportions (pi).
- "... which further minimizes the objective value" -- it minimizes a term that you introduced which is constant with respect to your learnable parameters. This is not a selling point, and I'm not sure why you bothered mentioning it.
- There's no mention of the substitution of log (1 - D(x)) for -log(D(x)) and its effect on the interpretation as a Jensen-Shannon divergence (which I'm not sure was quite right in the first place)
- Section 4: does the DAE introduced in DFM really introduce that much of a computational burden? 
- "Symmetric Kullback Liebler divergence" is not a well-known measure. The standard KL is asymmetric. Please define it.
- Figure 2 is illegible in grayscale.
- Improved-GAN score in Table 1 is misleading, as this was their no-label baseline. It's fine to include it but indicate it as such.
Update: many of my concerns were adequately addressed, however I still feel that calling this an avenue to "overcome mode collapse" is misleading. This seems aimed at improving coverage of the support of the data distribution; test log likelihood bounds via AIS (there are GAN baselines for MNIST in the Wu et al manuscript I mentioned) would have been more compelling quantitative evidence. I've raised my score to a 5.