Conditional Generative Adversarial Networks (cGANs) are finding increasingly widespread use in many application domains. Despite outstanding progress, quantitative evaluation of such models often involves multiple distinct metrics to assess different desirable properties, such as image quality, conditional consistency, and intra-conditioning diversity. In this setting, model benchmarking becomes a challenge, as each metric may indicate a different "best" model. In this paper, we propose the Fréchet Joint Distance (FJD), which is defined as the Fréchet distance between joint distributions of images and conditioning, allowing it to implicitly capture the aforementioned properties in a single metric. We conduct proof-of-concept experiments on a controllable synthetic dataset, which consistently highlight the benefits of FJD when compared to currently established metrics. Moreover, we use the newly introduced metric to compare existing cGAN-based models for a variety of conditioning modalities (e.g. class labels, object masks, bounding boxes, images, and text captions). We show that FJD can be used as a promising single metric for cGAN benchmarking and model selection.
Summary:
This paper extends the Fréchet Inception distance (FID) to the conditional distribution. To this end, the authors use an additional embedding for the condition variables (class, image, text), and concatenate to the data embedding. The proposed metric, name the Fréchet joint distance (FJD), captures three desired properties of conditional generative models: sample quality, conditional consistency, and sample diversity. The authors demonstrate that the proposed metric indeed captures the properties using a synthetic (dSprite) dataset, and shows reasonable values for real datasets.
Pros:
- FJD is an intuitive extension of FID for conditional generative models.
- FJD can be applied to various types of conditions (e.g., image and text), which cannot be done by prior work (e.g., [1]).
- The paper is easy to read and experimental details are clearly stated.
Cons:
1. FJD is a straightforward extension of FID.
FJD simply follows the FID formula but concatenates the condition embedding to the original data embedding. It is a straightforward extension of FID and suffers from the design choice problems due to the concatenation, as stated below.
2. FJD requires many design choices and not theoretically justified.
As FJD requires an additional embedding function h, balancing parameter \alpha, and merging function g, it raises a burden of design choices. While the authors give some suggestions, they are not theoretically justified. Also, one may use the statistical distances [2] between data distribution p_data(x,c) and model distribution p_g(x,c) to evaluate conditional generative models in a principled way, e.g., measure the KL-divergence using the density ratios [3]. The advantage of FJD over such metrics is unclear, as stated below.
3. The advantage over the prior work is not clear.
FJD and FID show the same trend in all reported experiments (Table 2, 3, 4), hence the advantage of FJD is unclear. Also, one may measure the FID score on conditional distributions, i.e., \sum_c FID( p_data(x|c), p_g(x|c) ). It also captures the desired three properties and would be a strong baseline for FJD. Besides, while the authors aim to design a single metric to stand the models in a line, identifying the trade-offs of models may also be useful. For example, Improved PRD [4] provides the precision-recall trade-offs of generative models, which provides some insights for the models.
[1] Ravuri and Vinyals. Classification Accuracy Score for Conditional Generative Models. NeurIPS 2019.
[2] https://en.wikipedia.org/wiki/Statistical_distance
[3] Uehara et al. Generative Adversarial Nets from a Density Ratio Estimation Perspective. arXiv 2016.
[4] Kynkäänniemi et al. Improved Precision and Recall Metric for Assessing Generative Models. NeurIPS 2019.