Generative Adversarial Networks (GANs) are an elegant mechanism for data generation. However, a key challenge when using GANs is how to best measure their ability to generate realistic data. In this paper, we demonstrate that an intrinsic dimensional characterization of the data space learned by a GAN model leads to an effective evaluation metric for GAN quality. In particular, we propose a new evaluation measure, CrossLID, that assesses the local intrinsic dimensionality (LID) of input data with respect to neighborhoods within GAN-generated samples. In experiments on 3 benchmark image datasets, we compare our proposed measure to several state-of-the-art evaluation metrics. Our experiments show that CrossLID is strongly correlated with sample quality, is sensitive to mode collapse, is robust to small-scale noise and image transformations, and can be applied in a modelfree manner. Furthermore, we show how CrossLID can be used within the GAN training process to improve generation quality.
Statistics based on KNN distances are ubiquitous in machine learning. In this paper the authors propose to apply the existing LID metric to GANs. The metric can be decomposed as follows: (1) Given a point x in X, compute the k-nearest neighbors KNN(x, X) and let those distances be R1, R2, …, Rk. Now, rewrite LID(x, X) = [max_over_i (log Ri) - mean_over_i (log Ri)] to uncover that the distribution of (log-)distances is summarized as a function of the max distance and the mean distance. (2) To extend the metric to two sets, A and B, define CrossLID(A; B) = E_(x in A) [LID(x, B)]. To see why CrossLID is useful, let X be the observed data and G the generated data. First consider CrossLID(A, B) where A=B=X which determines a lower-bound which is essentially the average (over elements of A) LID statistic determined by the underlying KNN graph of X. Now, keep A=X, and progressively change B to G (say by replacing some points from X with some points from G). This will induce a change of the distance statistics of some points from A, which will be detected on the individual LID scores of those points, and will hence be propagated to CrossLID. As a result, LID close to the baseline LID detects both sample quality issues as well as mode dropping/collapse issues. In practice, instead of computing this measure in the pixel space, one can compute it in the feature space of some feature extractor, or in some cases directly in the learned feature space of the generator. Finally, given some labeling of the points, one can keep track of the CrossLID statistic for each mode and use this during training to oversample modes for which the gap between the expected CrossLID and computed one is large.
Clarity: I think the clarity can be improved -- instead of stating the (rather abstract) properties of LID, the readers might benefit from the direct discussion of the LID estimator and a couple of examples, derive the max - mean relationship for the MLE estimator and provide some guiding comments. In a later section one might discuss why the estimator is so powerful and generally applicable. Secondly, the story starts with “discriminability of the distance measure” and the number of latent variables needed to do it, but I felt that this only complicated matters as many of these concepts are unclear at this point. 
Originality: Up to my knowledge, the proposed application is novel, albeit built on an existing (well-known) estimator. Nevertheless, the authors have demonstrated several desirable properties which might be proven useful in practice.
Significance of this work: The work is timely and attempts to address a critical research problem which hinders future research on deep generative models.
Pro:
- Generally well written paper, although the clarity of exposition can be improved. 
- Estimator is relatively easy to compute in practice (i.e. the bottleneck will still be in the forward and backward passes of the DNNs).
- Can be exploited further when labeled data is available
- Builds upon a strong line of research in KNN based estimators.
- Solid experimental setup with many ablation studies.
Con:
- FID vs CrossLID: I feel that many arguments against FID are too strong. In particular, in “robustness to small input noise” and “robustness to input transformation” you are changing the underlying distribution *significantly* -- why should the score be invariant to this? After all, it does try to capture the shift in distribution. In the robustness to sample size again FID is criticized to have a high-variance in low-sample size regime: This is well known, and that’s why virtually all work presenting FID results apply 10k samples and average out the results over random samples. In this regime it was observed that it has a high bias and low variance (Figure 1 in [1]). In terms of the dependency of the scores to an external model, why wouldn’t one compute FID on the discriminator feature space? Similarly, why wouldn’t one compute FID in the pixel space and get an (equally bad) score as LID in pixel space? Given these issues, in my opinion, Table 1 overstates the concerns with FID, and understates the issues with CrossLID. 
- FID vs CrossLID in practice: I argue that the usefulness comes from the fact that relative model comparison is sound. From this perspective it is critical to show that the Spearman’s rank correlation between these two competing approaches on real data sets is not very high -- hence, there are either sample quality or mode dropping/collapsing issues detected by one vs the other. Now, Figure 1 in [1] shows that this FID is sensitive to mode dropping. Furthermore, FID is also highly correlated with sample quality (page 7 of [2]).
- A critical aspect here is that in pixel space of large dimension the distances will tend to be very similar, and hence all estimators will be practically useless. As such, learning the proper features space is of paramount importance. In this work the authors suggest two remedies: (1) Compute a feature extractor by solving a surrogate task and have one extractor per data set. (2) During the training of the GAN, the discriminator is “learning” a good feature space in which the samples can be discriminated. Both of these have significant drawbacks. For (1) we need to share a dataset-specific model with the community. This is likely to depend on the preprocessing, model capacity, training issues, etc.. Then, the community has to agree to use one of these. On the other hand, (2) is only useful for biasing a specific training run. Hence, this critical aspect is not addressed and the proposed solution, while sensible, is unlikely to be adopted.
- Main contributions section is too strong -- avoiding mode collapse was not demonstrated. Arguably, given labeled data, the issue can be somewhat reduced if the modes correspond to labels. Similarly, if the data is well-clusterable one can expect a reduction of this effect. However, as both the underlying metric as well as the clustering depends on the feature space, I believe the claim to be too strong. Finally, if we indeed have labels or some assumptions on the data distribution, competing approaches might exploit it as well (as done with i.e. conditional GANs).
- In nonparametric KNN based density estimation, one often uses statistics based on KNN distances. What is the relation to LID?
With respect to the negative points above, without having a clear cut case why this measure outperforms and should replace FID in practice, I cannot recommend acceptance as introducing yet another measure might slow down the progress. To make a stronger case I suggest:
(1) Compute Spearman's rank correlation between FIDs and CrossLIDs of several trained models across these data sets.
(2) Compute the Pearson's correlation coefficient across the data sets. Given that your method has access to dataset specific feature extractors I expect it perform significantly better than FID.
 
[1] https://arxiv.org/pdf/1711.10337.pdf
[2] https://arxiv.org/pdf/1806.00035.pdf
========
Thank you for the detailed responses. I have updated my score from 5 to 6.