title: Rethinking FID: Towards a Better Evaluation Metric for Image Generation
summary: As with many machine learning problems, the progress of image generation
  methods hinges on good evaluation metrics. One of the most popular is the
  Frechet Inception Distance (FID). FID estimates the distance between a
  distribution of Inception-v3 features of real images, and those of images
  generated by the algorithm. We highlight important drawbacks of FID:
  Inception's poor representation of the rich and varied content generated by
  modern text-to-image models, incorrect normality assumptions, and poor sample
  complexity. We call for a reevaluation of FID's use as the primary quality
  metric for generated images. We empirically demonstrate that FID contradicts
  human raters, it does not reflect gradual improvement of iterative
  text-to-image models, it does not capture distortion levels, and that it
  produces inconsistent results when varying the sample size. We also propose an
  alternative new metric, CMMD, based on richer CLIP embeddings and the maximum
  mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased
  estimator that does not make any assumptions on the probability distribution of
  the embeddings and is sample efficient. Through extensive experiments and
  analysis, we demonstrate that FID-based evaluations of text-to-image models may
  be unreliable, and that CMMD offers a more robust and reliable assessment of
  image quality.
paper_authors._id: 65a9e617e0ee7990a6ed4e3c
paper_authors.name: Sadeep Jayasumana
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:43:07.027Z
paper_authors._id: 65393b586882a4dcf7bba384
paper_authors.avatarUrl: /avatars/fb8e808b3b526cc9af4ac625b1935088.svg
paper_authors.fullname: Sadeep Jayasumana
paper_authors.type: user
paper_authors.user: sadeepj
paper_authors._id: 65a9e617e0ee7990a6ed4e3d
paper_authors.name: Srikumar Ramalingam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-19T13:43:13.050Z
paper_authors._id: 64916f233da8270af8580eb1
paper_authors.avatarUrl: /avatars/40100ef7f72c8233f40713264bf39fc5.svg
paper_authors.fullname: Srikumar Ramalingam
paper_authors.type: user
paper_authors.user: srikumar
paper_authors._id: 65a9e617e0ee7990a6ed4e3e
paper_authors.name: Andreas Veit
paper_authors._id: 65a9e617e0ee7990a6ed4e3f
paper_authors.name: Daniel Glasner
paper_authors._id: 65a9e617e0ee7990a6ed4e40
paper_authors.name: Ayan Chakrabarti
paper_authors._id: 65a9e617e0ee7990a6ed4e41
paper_authors.name: Sanjiv Kumar
paper_title: Rethinking FID: Towards a Better Evaluation Metric for Image Generation
paper_summary: As with many machine learning problems, the progress of image generation
  methods hinges on good evaluation metrics. One of the most popular is the
  Frechet Inception Distance (FID). FID estimates the distance between a
  distribution of Inception-v3 features of real images, and those of images
  generated by the algorithm. We highlight important drawbacks of FID:
  Inception's poor representation of the rich and varied content generated by
  modern text-to-image models, incorrect normality assumptions, and poor sample
  complexity. We call for a reevaluation of FID's use as the primary quality
  metric for generated images. We empirically demonstrate that FID contradicts
  human raters, it does not reflect gradual improvement of iterative
  text-to-image models, it does not capture distortion levels, and that it
  produces inconsistent results when varying the sample size. We also propose an
  alternative new metric, CMMD, based on richer CLIP embeddings and the maximum
  mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased
  estimator that does not make any assumptions on the probability distribution of
  the embeddings and is sample efficient. Through extensive experiments and
  analysis, we demonstrate that FID-based evaluations of text-to-image models may
  be unreliable, and that CMMD offers a more robust and reliable assessment of
  image quality.
paper_ai_summary: The paper identifies and critiques limitations of FID as an evaluation metric for image generation, proposing a new metric, CMMD, based on CLIP embeddings and maximum mean discrepancy for more reliable evaluations.
paper_ai_keywords: Frechet Inception Distance Inception-v3 text-to-image models CLIP embeddings maximum mean discrepancy Gaussian RBF kernel