This paper provides theoretical analysis of generative adversarial networks (GANs) to explain its advantages over other standard methods of learning probability measures. GANs learn a probability through observations, using the objective function with a generator and a discriminator. While many empirical results indicate that GANs can generate realistic samples, the reason for such successful performance remains unelucidated. This paper focuses the situation where the target probability measure satisfies the disconnected support property, which means a separate support of a probability, and relates it with the advantage of GANs. It is theoretically shown that, unlike other popular models, GANs do not suffer from the decrease of generalization performance caused by the disconnected support property. We rigorously quantify the generalization performance of GANs of a given architecture, and compare it with the performance of the other models. Based on the theory, we also provide a guideline for selecting deep network architecture for GANs. We demonstrate some numerical examples which support our results.
This paper provides a theoretical study of GANs in the following setting:
- The target distribution has a locally smooth density on a compact set [0, 1]^D. It might be supported only on M disjoint components, each of which has a smooth boundary, within that compact set.
- The latent noise dimension (inputs to the generator) is of the same dimension as the data.
- An IPM loss (1) is used.
- The discriminator functions of the IPM and the generator networks are both ReLU networks of at most L layers and at most S total nonzero weights, with all weights having magnitude at most B.
- The discriminator functions are Lipschitz continuous; this is implied by the previous assumption, but the bound is tighter if we have a tighter constraint here.
- We obtain the generator which exactly minimizes the IPM between the empirical distributions of m samples from the model and n samples from the target.
- Maybe: there is some g such that P* is produced by g. It's not clear in your statement of Theorem 1 whether this is necessary, or exactly what it means when you say that in Lemma 3, and I haven't fully checked what's used yet; it would help to explicitly say this is not assumed if it's not.
Pointing out that the kinds of distributions handled by GANs often have disjoint support, and analyzing this case, is certainly of interest.
The full-dimensional support assumption is not ideal, but of course just because the analysis doesn't apply to most practical GAN settings doesn't mean it's not an important step towards one that does.
Also note that the assumption about the structure of the networks eliminates the MMD GANs that you use in experiments -- which have a kernel function at the top of the critic network -- though it does allow for most GAN variants. Maybe the most interesting algorithm for this setting is the Coulomb GAN (ICLR 2018, https://arxiv.org/abs/1708.08819 ), which uses a neural network critic of the kind you study but estimates a distance which (unlike the Wasserstein and most other GAN objective functions) has good statistical convergence properties (kappa=2 as you mention).
My biggest concern by far, though, is Proposition 1. You present it as if it's a lower bound: establishing that there is some class of distributions for which Theorem 1 shows that GANs can account for local smoothness and standard methods are shown not to be able to. This isn't what you do; instead you exhibit a class of distributions for which Theorem 1 shows that GANs can account for local smoothness, and previous analyses of standard methods do not show that they are able to take advantage of it. This is not the same thing at all! Although the previous upper bounds have matching lower bounds, you don't demonstrate (and it is likely not the case) that the distribution you show fits into the class of distributions used by the previous lower bounds, and so it remains very possible that other methods are able to take advantage of local smoothness as well as GANs do.
Given that Proposition 1 is hence a very weak statement, your main contribution in terms of disconnected support becomes "we can show that GANs can adapt to disconnected supports (under these other assumptions) that has not previously been shown for other methods." Not only is this a weaker result, but the degree to which you show GANs can take advantage of local smoothness is somewhat limited: at least with the parameter choices in Corollary 1, the smoothness only improves m dependence, not n dependence.
But in the GAN setting, m is essentially a question of how long you optimize for (and the relative rates between generator and discriminator updates, and various other questions like that out of scope for this paper), not any kind of externally fixed limitation like n. It's perhaps not too surprising that you don't show that optimizing a WGAN is statistically easier than estimating Wasserstein the distance, but given that estimating Wasserstein is so hard in high dimensions, it's a little disappointing. (Maybe it's easier, so that kappa is smaller, with locally smooth densities as here, though I don't know of any results like that offhand.)
(It's interesting, then, that in Corollary 1 the generator complexity depends only on m, basically the amount of optimization you're willing to do, while the discriminator complexity depends only on the available number of target samples n.)
Another concern is that to me, it is not very clear exactly what the statements of the assumptions mean. For example, does Theorem 1 apply only if I search over all generators in the class \mathcal G = \Xi(S_g, B_g, L_g)? In particular, does this mean that I have to consider all possible architectures matching those constraints, including allowing for all possible depths up to L_g, and all possible ways of allocating widths of the various interior layers / which weight entries are fixed at zero? It seems so, but this could be more explicit in the statements, even just by replacing "an existing \mathcal G" with "\mathcal G = \Xi(S_g, B_g, L_g)" in the statement of (e.g.) Theorem 1.
In your numerical experiments: you don't make it at all clear enough that you're plotting *different loss functions* for the GANs and the other methods! (You say this, but only in the text where it should definitely also be in the figure caption.) What happens if you plot the L2 difference for all methods, and the MMD/Wasserstein for all methods? Looking at Figure 6, it's not obvious that the GAN would do so substantially better. (It does seem to perhaps have the overall scale of the two components better than the other methods, but it doesn't look like as enormous a difference as it seems from Figure 5.)
Overall: Theorem 1 is of interest, but the results and especially the comparison to classic methods are not as resounding as they're presented here.
(Note that I have not (yet) verified or even really read most of the proofs; I might come back and do that later.)
Smaller points:
- I don't think that f-GAN actually fits in the framework (1) as you claim, since it needs to use the conjugate of f on the samples from Q. Also, the original GAN does fit into (1) but not your assumptions about the network form of f, since it needs a log in its activations.
- Another class of generative models where disconnected supports are really important is normalizing flows, which often build "bridges" between separated modes because (like your Lemma 2) their generators are constrained to be smooth and invertible. See e.g. Figure 2 of https://arxiv.org/abs/1810.01367 (who propose a new normalizing flow less susceptible to these problems). 
- Remark 1 seems so obvious that it need not even be stated, since beta-smooth implies beta'-smooth for beta' < beta. It would only be interesting if you could actually take advantage of the smoother components somehow.
- Many papers in your bibliography are cited only as arXiv preprints when they were actually published in various places. For example, the first four papers were published at ICLR 2017, ICML 2017, ICML 2017, and ICLR 2018, respectively.
- There are many small typos and grammatical errors in the draft, including some that would be caught by a spell-checker ("methdos" on page 8), and an undefined LaTeX reference at the top of page 2. It would benefit from a thorough proofread.