We define a goodness of fit measure for generative networks which captures how well the network can generate the training data, which is necessary to learn the true data distribution. We demonstrate how our measure can be leveraged to understand mode collapse in generative adversarial networks and provide practitioners with a novel way to perform model comparison and early stopping without having to access another trained model as with Frechet Inception Distance or Inception Score. This measure shows that several successful, popular generative models, such as DCGAN and WGAN, fall very short of learning the data distribution. We identify this issue in generative models and empirically show that overparameterization via subsampling data and using a mixture of models improves performance in terms of goodness of fit.
This work proposed a new goodness of fit measure for generative network evaluations, which is based on how well the network can generate the training data. The measure is zero if the network could perfectly recover the training data, and would represent how far it is from generating the training set in the average manner of the total least square sense, where the one-to-one mapping between the generated data and the training sample is constructed through latent space optimization. Using the proposed measure, the authors showed an interesting trend present in the DCGAN training and the impact of the residual connection. The authors might want to add some discussion in Section 4.2 regarding why the residual connection is detrimental for covering the support.  Increasing the model complexity through larger latent space dimension and learning mixtures is proposed as solutions to improve the measure as well.
With all the interesting results presented, I still have the concerns about the sensitivity of the proposed measure:
- It is an average over the training data or the selected sample. Above Section 4, the authors argued that "\hat{F}(G) > 0 meaning that we do not observe any memorization". This seems overly assertive. Since the measure is an average over the training data, it has difficulty to differentiate between one network which has almost zero value for part of the training data but large values for the rest, and another network with roughly the same \hat{F}(G) value but small values for all training data. The variance could help, but can not resolve this issue. This would be more important when the training data contains noise or outliers.
- It only concerns the generation of the training data, but not the sampled data from the network (at least not directly). Therefore it has no direct control of the fidelity of the generated samples.
- As shown by the authors, the proposed measure can be considered as the approximation of the true probability support not covered by the generative models, which also defines a necessary condition to avoid mode collapse. But what about the other part? It would have difficulty comparing two models with the same support but different high-density areas. Indeed, there are existing works which consider both the precision and recall of the generative models [1, 2, 3], and directly work with the generated samples instead of the training data. These should be discussed and compared with, not just the FID scores which have already been shown to have issues [3]. 
Some notations:
- In the last equation on Page 2,  should it be L_{G} instead of L_{D}?
- In the first equation on Page 3, should the denominator be N_{B} instead of N_{N}?
- "Optimality" in terms of generative models may depend on the downstream tasks. I do not think there exists a universal definition of "optimality" for generative models.
[1] M.S.M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. NeurIPS 2018.
[2] L. Simon, R. Webster, and J. Rabin. Revisiting precision and recall definition for generative model evaluation. ICML 2019.
[3] T. Kynkaanniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. Arxiv:1904.06991.