Disconnected Manifold Learning for Generative Adversarial Networks
Natural images may lie on a union of disjoint manifolds rather than one globally connected manifold, and this can cause several difficulties for the training of common Generative Adversarial Networks (GANs). In this work, we first show that single generator GANs are unable to correctly model a distribution supported on a disconnected manifold, and investigate how sample quality, mode dropping and local convergence are affected by this. Next, we show how using a collection of generators can address this problem, providing new insights into the success of such multi-generator GANs. Finally, we explain the serious issues caused by considering a fixed prior over the collection of generators and propose a novel approach for learning the prior and inferring the necessary number of generators without any supervision. Our proposed modifications can be applied on top of any other GAN model to enable learning of distributions supported on disconnected manifolds. We conduct several experiments to illustrate the aforementioned shortcoming of GANs, its consequences in practice, and the effectiveness of our proposed modifications in alleviating these issues.
After the rebuttal and discussions with the other reviewers I increased my overall rating from 4 to 5. I agree with the authors and the other reviewers that the observation that a disconnected data manifold can negatively affect GAN performance is important. However, I think the authors should have conducted more experiments to investigate if this problem really arises in practice (e.g. using lower dimensional embeddings in some feature space). I also disagree with the authors that the MSD measure in Euclidean space is a good measure, as Euclidean distance is known to be rather meaningless for images. Moreover, the experimental validation should *always* include the simple baseline where the generators are all trained jointly without the InfoGAN-loss. In the rebuttal, the authors show that this baseline performs (slightly) worse than the proposed method for the toy example, but I think it's vital to include this baseline in all experiments (especially in the celebA + LSUN experiment). I also think it is important to disentangle the effect of the increased model complexity of the proposed method from the effect of the regularization.  ==================================================================  # Summary This paper discusses the potential limitation of current state-of-the-art generative models in modeling probability distributions whose support is concentrated on a union of disjoint submanifolds. To circumvent this problem, the authors propose to use a simple variant of InfoGAN [1] with discrete variables.  # Significance The observation that real-world data distribution might be concentrated on a union of disjoint submanifolds  is important and hints at a possible limitation of current state-of-the-art models. Unfortunately, the authors do not directly verify this hypothesis but instead use it to motivate their method which appears to be identical to InfoGAN [1] with discrete latent variables (+ a learnable prior). It is also not clear why the authors need the InfoGAN architecture to get a good generative model and why the same effect could not be achieved by simply including discrete latent variables into the model (the explanation in ll. 113-114 is not really satisfying as it is not verified experimentally).  # Clarity The paper is well-written and easy to follow. Including an example in ll. 17-18 is a nice start for the paper. However, this particular example is not really a good one, as it is certainly possible to transform a tree into a bird, e.g. by simply "zooming" into the tree where a bird sits.  # Experiments The experiments are interesting, but the evaluation is a bit unclear. Was the MSD computed in Euclidean space or Feature space (ll. 176-185)? Euclidean space is probably not a good choice for an evaluation metric here, as the Euclidean distance between images is often meaningless. The discussion in ll. 192-202 why overfitting cannot occur is also not convincing, as it is not experimentally verified. A good evaluation metric should certainly be able to detect overfitting. The hypothesis that real world data distributions often contain multiple disconnected components should also be verified experimentally, e.g. by using lower dimensional embeddings of the distribution. The experiment on the union of celebA and LSUN bedrooms is interesting, but also a bit artificial, as here it is very clear that this particular probability distribution has multiple disjoint components. How do similar experiments on celebA and/or LSUN alone look like?   # Other comments / questions * There is an extra "." in l. 11 * "no definite reason" in l. 35 should probably be replaced with "no satisfying explanation"  * The discussion in ll. 99-104 is interesting, but when the generator cannot model the support of the true data distribution the realizability assumption of [2,3] also breaks down, which would probably be an even more compelling argument * The explanation that we need to "encourage these generators to each focus on a different submanifold of the real data" (ll. 113-114) is not really satisfying, as this could simply be learned by the generators. There are also no experiments in this paper that verify this claim. * The relationship between "partially covered real submanifolds" and "mode collapse" (l. 114) is completely unclear. * "corss entropy" should read "cross entropy" (l. 160)  [1] Chen, Xi, et al. "Infogan: Interpretable representation learning by information maximizing generative adversarial nets." Advances in neural information processing systems. 2016. [2] Nagarajan, Vaishnavh, and J. Zico Kolter. "Gradient descent GAN optimization is locally stable." Advances in Neural Information Processing Systems. 2017. [3] Mescheder, Lars, Andreas Geiger, and Sebastian Nowozin. "Which Training Methods for GANs do actually Converge?." International Conference on Machine Learning. 2018.