GANs have shown how deep neural networks can be used for generative modeling, aiming at achieving the same impact that they brought for discriminative modeling. The first results were impressive, GANs were shown to be able to generate samples in high dimensional structured spaces, like images and text, that were no copies of the training data. But generative and discriminative learning are quite different. Discriminative learning has a clear end, while generative modeling is an intermediate step to understand the data or generate hypothesis. The quality of implicit density estimation is hard to evaluate, because we cannot tell how well a data is represented by the model. How can we certainly say that a generative process is generating natural images with the same distribution as we do? In this paper, we noticed that even though GANs might not be able to generate samples from the underlying distribution (or we cannot tell at least), they are capturing some structure of the data in that high dimensional space. It is therefore needed to address how we can leverage those estimates produced by GANs in the same way we are able to use other generative modeling algorithms.
The main take-away messages of this paper seem to be:
1. GANs don't really match the target distribution. Some previous theory supports this, and some experiments are provided here demonstrating that the failure seems to be largely in under-sampling the tails, and sometimes perhaps in introducing spurious modes.
2. Even if GANs don't exactly match the target distribution, their outputs might still be useful for some tasks.
(I wouldn't be surprised if you disagree with what the main takeaways are; I found the flow of the paper somewhat disjointed, and had something of a hard time identifying what the "point" was.)
Mode-dropping being a primary failure mode of GANs is already a fairly accepted hypothesis in the community (see, e.g. Mode Regularized GANs, Che et al ICLR 2017, among others), though some extra empirical evidence is provided here.
The second point is, in my opinion, simultaneously (i) an important point that more GAN research should take to heart, (ii) relatively obvious, and (iii) barely explored in this paper. The only example in the paper of using a GAN for something other than directly matching the target distribution is PassGAN, and even that is barely explored beyond saying that some of the spurious modes seem like reasonable-ish passwords.
Thus though this paper has some interesting aspects to it, I do not think its contributions rise to the level required for an ICLR paper.
Some more specifics:
Section 2.1 discusses four previous theoretical results about the convergence of GANs to the true density. This overview is mostly reasonable, and the discussion of Arora et al. (2017) and Liu et al. (2017) do at least vaguely support the conclusion in the last section of this paragraph. But this section is glaringly missing an important paper in this area: Arjovsky and Bottou (2017), cited here only in passing in the introduction, who proved that typical GAN architectures *cannot* exactly match the data distribution. Thus the question of metrics for convergence is of central importance, which it seems should be important to the topic of the present paper. (Figure 3 of Danihelka et al. https://arxiv.org/abs/1705.05263 gives a particularly vivid example of how optimizing different metrics can lead to very different results.) Presumably different metrics lead to models that are useful for different final tasks.
Also, although they do not quite fit into the framing of this section, Nowozin et al.'s local convergence proof and especially the convergence to a Nash equilibrium argument of Heusel et al. (NIPS 2017, https://arxiv.org/abs/1706.08500) should probably be mentioned here.
The two sample testing section of this paper, discussed in Section 2.2 and then implemented in Section 3.1.1, seems to be essentially a special case of what was previously done by Sutherland et al. (2017), except that it was run on CIFAR-10 as well. However, the bottom half of Table 1 demonstrates that something is seriously wrong with the implementation of your tests: using 1000 bootstrap samples, you should reject H_0 at approximately the nominal rate of 5%, not about 50%! To double-check, I ran a median-heuristic RBF kernel MMD myself on the MNIST test set with N_test = 100, repeating 1000 times, and rejected the null 4.8% of the time. My code is available at https://gist.github.com/anonymous/2993a16fbc28a424a0e79b1c8ff31d24 if you want to use it to help find the difference from what you did. Although Table 1 does indicate that the GAN distribution is more different from the test set than the test set is from itself, the apparent serious flaw in your procedure makes those results questionable. (Also, it seems that your entry labeled "MMD" in the table is probably n * MMD_b^2, which is what is computed by the code linked to in footnote 2.)
The appendix gives a further study of what went wrong with the MNIST GAN model, arguing based on nearest-neighbors that the GAN model is over-representing modes and under-representing the tails. This is fairly interesting; certainly more interesting than the rehash of running MMD tests on GAN outputs, in my opinion.
Minor:
In 3.1.1, you say "ideally the null hypothesis H0 should never be rejected" – it should be rejected at most an alpha portion of the time.
In the description of section 3.2, you should clarify whether the train-test split was done such that unique passwords were assigned to a single fold or not: did 123456 appear in both folds? (It is not entirely clear whether it should or not; both schemes have possible advantages for evaluation.)