APPROXIMABILITY OF DISCRIMINATORS IMPLIES DIVERSITY IN GANS
While Generative Adversarial Networks (GANs) have empirically produced impressive results on learning complex real-world distributions, recent works have shown that they suffer from lack of diversity or mode collapse. The theoretical work of Arora et al. (2017a)  suggests a dilemma about GANs' statistical properties: powerful discriminators cause overfitting, whereas weak discriminators cannot detect mode collapse. By contrast, we show in this paper that GANs can in principle learn distributions in Wasserstein distance (or KL-divergence in many cases) with polynomial sample complexity, if the discriminator class has strong distinguishing power against the particular generator class (instead of against all possible generators). For various generator classes such as mixture of Gaussians, exponential families, and invertible and injective neural networks generators, we design corresponding discriminators (which are often neural nets of specific architectures) such that the Integral Probability Metric (IPM) induced by the discriminators can provably approximate the Wasserstein distance and/or KL-divergence. This implies that if the training is successful, then the learned distribution is close to the true distribution in Wasserstein distance or KL divergence, and thus cannot drop modes. Our preliminary experiments show that on synthetic datasets the test IPM is well correlated with KL divergence or the Wasserstein distance, indicating that the lack of diversity in GANs may be caused by the sub-optimality in optimization instead of statistical inefficiency.
[pros]
This paper proposes the notion of restricted approximability, and provides a sample complexity bound, polynomial in the dimension, for GANs.
The proposal is especially useful in investigating possible cause of the lack of diversity in GANs.
[cons]
Whether it proposes use of properly-designed discriminator architecture in GAN learning is not clear enough.
The claimed ability of the proposed method to avoid mode collapse is not directly addressed in the experiments presented in the appendices.
[quality]
The contents of Section 3 may be useful as case studies but are not used in the following sections on neural network generators. It would thus be better to include experimental results into the main part of the paper rather than the current contents in Section 3.
[clarity]
In most parts of this paper, the authors seem to propose designing a proper discriminator architecture according to the generator class, and the discriminator architecture is to be used in GAN learning. It seems, however, that a "properly-designed discriminator architecture" is not used at all in the experiments in Appendix F. A comparison between a "properly-designed discriminator architecture" and a "vanilla fully-connected distriminator" is found in Appendix G.4, where the advantage of the former seems marginal. The authors also seem to use the proposal not to improve GAN learning but rather as a tool for evaluation, in order to see whether the lack of diversity in GANs comes either from failure of properly evaluating the Wasserstein distance or from insufficient optimization in learning. These two distinct subjects are discussed in a mixed way, which reduces clarity of the presentation.
In the experiments in Appendix G, it is claimed that a discriminator with the architecture specified in Lemma 4.1 is used in GAN learning, but either weight clamping or gradient penalty is used as well. It is unclear how the specifications in Lemma 4.1 for the parameter $\phi$ are combined with weight clamping or gradient penalty.
Some statements include forward reference, which obscure readability. For example, in the last paragraph of Section 1.1 "the statistical properties of GANs" are mentioned without an explicit statement as to what they mean, which are given later in page 3, lines 6-12. As another example, in the third paragraph of Section 1.3 the authors start discussing the KL-divergence, but at this point it is not evident at all why they do it. It is not until Section 4.1 that the reader can understand the reason by observing that the main theorem (Theorem 4.2) is proved by making use of KL-divergence.
[originality]
The idea of introducing the notion of restricted approximability and discussing a sample complexity bound, polynomial in the dimension, for GANs are considered original.
[significance]
The whole arguments in this paper are based on the assumption that both $p$ and $q$ are in the class $\mathcal{G}$. In the context of GAN learning, it poses no problem for the generator since we explicitly parameterize it, for example using a neural network, but in practice there is no guarantee that the target distribution also belongs to the same class, and this point would affect significance of the proposal. One may argue that when one employs a certain neural network architecture for the generator one expects that the target distribution is well expressed by a network with the prescribed architecture. But the question as to what will happen when the target distribution does not belong to the class $\mathcal{G}$ remains. In any case, no discussion is presented in this paper as for this question.
[minor points]
Page 3, line 45: for low-dimensional (dimensions -> distributions)
Page 4, line 8: Remove the parentheses enclosing Lopez-Paz & Oquab, 2016.
Page 4, lines 20-21: Duplicate parentheses.
Page 4, line 7: the true and estimated distribution(s) exist.
Page 5, line 33: the lower and upper bound(s) differ
Page 7, line 9: What do "some assumptions" refer to?
Page 8, line 44: The(re) exists a discriminator class
Page 19, line 1: there exi(s)ts a coupling