Answers.Body: <p>Consider that the problem of evaluating (or measuring) the quality of generated images, can be expressed in terms of <em>texture quality</em> and/or <em>perceptual quality</em>.</p>
  <ul>
  <li>The texture quality measures how close the generated texture is to the original one. Intuitively, measuring textures is equal to measuring pixel-level differences. For example you can achieve this by a <em>mean squared error</em> (MSE) over the single pixels.</li>
  <li>Perceptual metrics look instead at the (global) meaning of the image rather than at its individual pixel values. Therefore, perceptual quality aims to measure the agreement in semantics of two images, ideally reflecting the human understanding of them.</li>
  </ul>
  <p>Actually there is a trade-off between texture and perceptual quality, meaning that the same method can't maximize both. In general, VAEs are said to generate better looking textures while GANs to provide images with way better semantics: in other words, GANs generally generate plausible images but with inaccurate textures - indeed, for very big models the errors tend to narrow.</p>
  <blockquote>
  <p>Does this mean that images need to have similar properties to those in ImageNet?</p>
  </blockquote>
  <p>Well, the Inception score is defined from the activations of a pre-trained Inception-V3 so you're relying on what the model has learned. I mean it should be expected to better rank natural images rather than something that is radically different and never seen during training. As an alternative score you can look at the <a href="https://en.wikipedia.org/wiki/Fr%C3%A9chet_inception_distance" rel="nofollow noreferrer">Frechet Inception Distance</a> (FID) which also considers the distribution of the real images, which is expected to provide better evaluations in general.</p>
  <blockquote>
  <p>What is the best way to evaluate GAN-generated imagery for non-photographic data?</p>
  </blockquote>
  <p>It can be hard to answer but you can also try image processing metrics like the <a href="https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio" rel="nofollow noreferrer">PSNR</a> or the <a href="https://en.wikipedia.org/wiki/Structural_similarity" rel="nofollow noreferrer">SSIM</a>, as alternatives to IS and/or FID. These work for general kind of images, and you may want to try both in order to understand which better aligns with your notion of &quot;better&quot; quality. In general, the SSIM should be superior to both MSE and PSNR since it's designed to better match the human perception of images.</p>
  <ul>
  <li>Implementations (tensorflow): <a href="https://www.tensorflow.org/api_docs/python/tf/image/ssim" rel="nofollow noreferrer">ssim</a>, and <a href="https://www.tensorflow.org/api_docs/python/tf/image/psnr" rel="nofollow noreferrer">psnr</a>.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-23T13:11:53.177Z
Answers.Text: Maybe it was implied in this answer, but one could in theory finetune (or train from scratch) an Inception or shallower model on some task (self-supervised task, if no label are available), and then use it to calculate IS. It can be expensive indeed, but it is still a solution. I do not also exclude that one could find a pretrained model on similar data, e.g satellite imagery.
Answers.UserDisplayName: Ciodar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-12T07:58:55.083Z
Answers.Text: @abcd The Inception Score measures the image quality by measuring the confidence and the diversity of the predictions computed by a separate, pre-trained classifier. The hypothesis is that if images look real, the classifier should classify them with high confidence. IF the domain of generated images differs from the classifier's domain, images could have a much lower confidence. Say you're generating images of a *really* different domain (e.g. satellite data, medical data, MNIST), a classifier trained on ImageNet should not have high confidence, even if images are qualitatively very good.
Answers.UserDisplayName: Ciodar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-12T13:39:24.977Z
Answers.Text: @abcd I guess the main reason is to make sure their work is comparable with papers which relied only on IS
Answers.UserDisplayName: Ciodar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-20T08:33:41.790Z
Answers.LastActivityDate: 2023-06-20T08:33:41.790Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>Most online sources recommend using versions of the Inception score to evaluate the synthetic images generated by a GAN. These scores are pre-trained on the InceptionV3 model. Does this mean that images need to have similar properties to those in ImageNet? My images have only one channel and are images of climate data so have very different properties. What is the best way to evaluate GAN-generated imagery for non-photographic data?</p>
  
Tags: <generative-adversarial-networks><testing><dc-gan><fid-score>
Title: What are alternatives to Inception Score? Can it be used for non-photographic image types?
UserDisplayName: ThreeOrangeOneRed
ThreadText: # What are alternatives to Inception Score? Can it be used for non-photographic image types?
  
  (Asked by: username_0 on 2023-06-19)
  
  Most online sources recommend using versions of the Inception score to evaluate the synthetic images generated by a GAN. These scores are pre-trained on the InceptionV3 model. Does this mean that images need to have similar properties to those in ImageNet? My images have only one channel and are images of climate data so have very different properties. What is the best way to evaluate GAN-generated imagery for non-photographic data?
  
  **Tags:** generative-adversarial-networks, testing, dc-gan, fid-score
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-20. Score: 2
  
  Consider that the problem of evaluating (or measuring) the quality of generated images, can be expressed in terms of *texture quality* and/or *perceptual quality*.
  
  * The texture quality measures how close the generated texture is to the original one. Intuitively, measuring textures is equal to measuring pixel-level differences. For example you can achieve this by a *mean squared error* (MSE) over the single pixels.
  * Perceptual metrics look instead at the (global) meaning of the image rather than at its individual pixel values. Therefore, perceptual quality aims to measure the agreement in semantics of two images, ideally reflecting the human understanding of them.
  
  Actually there is a trade-off between texture and perceptual quality, meaning that the same method can't maximize both. In general, VAEs are said to generate better looking textures while GANs to provide images with way better semantics: in other words, GANs generally generate plausible images but with inaccurate textures - indeed, for very big models the errors tend to narrow.
  
  > Does this mean that images need to have similar properties to those in ImageNet?
  
  Well, the Inception score is defined from the activations of a pre-trained Inception-V3 so you're relying on what the model has learned. I mean it should be expected to better rank natural images rather than something that is radically different and never seen during training. As an alternative score you can look at the [Frechet Inception Distance](https://en.wikipedia.org/wiki/Fr%C3%A9chet_inception_distance) (FID) which also considers the distribution of the real images, which is expected to provide better evaluations in general.
  
  > What is the best way to evaluate GAN-generated imagery for non-photographic data?
  
  It can be hard to answer but you can also try image processing metrics like the [PSNR](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio) or the [SSIM](https://en.wikipedia.org/wiki/Structural_similarity), as alternatives to IS and/or FID. These work for general kind of images, and you may want to try both in order to understand which better aligns with your notion of "better" quality. In general, the SSIM should be superior to both MSE and PSNR since it's designed to better match the human perception of images.
  
  * Implementations (tensorflow): [ssim](https://www.tensorflow.org/api_docs/python/tf/image/ssim), and [psnr](https://www.tensorflow.org/api_docs/python/tf/image/psnr).
  
  > **Comments:**
  > **username_2**: Maybe it was implied in this answer, but one could in theory finetune (or train from scratch) an Inception or shallower model on some task (self-supervised task, if no label are available), and then use it to calculate IS. It can be expensive indeed, but it is still a solution. I do not also exclude that one could find a pretrained model on similar data, e.g satellite imagery.  
  > **username_2**: @abcd The Inception Score measures the image quality by measuring the confidence and the diversity of the predictions computed by a separate, pre-trained classifier. The hypothesis is that if images look real, the classifier should classify them with high confidence. IF the domain of generated images differs from the classifier's domain, images could have a much lower confidence. Say you're generating images of a *really* different domain (e.g. satellite data, medical data, MNIST), a classifier trained on ImageNet should not have high confidence, even if images are qualitatively very good.  
  > **username_2**: @abcd I guess the main reason is to make sure their work is comparable with papers which relied only on IS  
  
  ---
  
  