Assessing Generative Models via Precision and Recall
Recent advances in generative modeling have led to an increased interest in the study of statistical divergences as means of model comparison. Commonly used evaluation methods, such as the Fréchet Inception Distance (FID), correlate well with the perceived quality of samples and are sensitive to mode dropping. However, these metrics are unable to distinguish between different failure cases since they only yield one-dimensional scores. We propose a novel definition of precision and recall for distributions which disentangles the divergence into two separate dimensions. The proposed notion is intuitive, retains desirable properties, and naturally leads to an efficient algorithm that can be used to evaluate generative models. We relate this notion to total variation as well as to recent evaluation metrics such as Inception Score and FID. To demonstrate the practical utility of the proposed approach we perform an empirical study on several variants of Generative Adversarial Networks and Variational Autoencoders. In an extensive set of experiments we show that the proposed metric is able to disentangle the quality of generated samples from the coverage of the target distribution.
This paper contributes some original thinkings on how to assess the quality of a generative model. The new evaluation metric, as defined by the distributional precision and recall statistics (PRD), overcomes a major drawback from prior-arts: that evaluation scores are almost exclusively scalar metrics. The author(s) attributes intuitive explanations to this new metric and experimentally reached a conclusion that it is able to disentangle the quality from the coverage, two critical aspects wrt the quality of a learned synthetic sampler. An efficient algorithm is also described and theoretically justified. The quality of this work seems okay, yet I am prone to a neutral-to-negative rating. This is mainly because I am not sure I fully understand the technical details from the paper after a few attempts, and I have doubts about its value to the future development of improved generative models. I will reconsider my decision if the author(s) can properly address my concerns detailed below.   * While I strongly concur with the fact that a comprehensive set of indicators are needed to properly evaluate the quality of a generative model, it is not so obvious to me why a curve-type metric like PRD proposed in the current study is more preferable. Such metrics can be more open to (subjective) interpretations, which I found undesirable. Because this might add to the confusion of what makes a good generative model.   * Overload of symbol \bar{\apha} and \bar{\beta}.   * When doing a comparison involving more than 3 distributions, shouldn't we use the pooled samples from all the models (and data) to construct the local bins? (As oppose to pair-wise matching for the evaluation of PRD score, which I assume is what the author(s) did in the current experiments).   * The author(s) kind of implies there is this trade-off between precision and recall, at least in practice. I need more proof on this point.   * As also mentioned by the author(s), KNN may not be a reliable tool as the cluster assignment might be arbitrary.   * Fig. 7 left, adding marginal histograms might serve to better interpret the observation.   * The experimental results in the main text seem a little seem somewhat weak, as all experiments are only based on simple binary/gray-scale images. More extensive set of experiments, like some representative toy models and more sophisticated image datasets, should be considered in the main text.   * This paper does not provide any insights on how to improve generative modeling wrt the novel metric discussed. And this is a major drawback.