Answers.Body: <p>I will answer my own question to try and provide some insights.</p>
  <p>My research supervisor suggested that I should use the <a href="https://www.cns.nyu.edu/%7Elcv/ssim/" rel="nofollow noreferrer">SSIM metric</a> or some other well-known image processing metric (see the book <a href="https://www.morganclaypool.com/doi/abs/10.2200/s00010ed1v01y200508ivm003" rel="nofollow noreferrer">&quot;Modern Image Quality Assessment&quot; by Wang and Bovik</a>) for assessing the visual similarity of an images.</p>
  <p>Another way I evaluate the performance of an autoencoder is by simply visually comparing the input and output images taken from the test set. This is by no means very scientific, but it gives a good idea whether an autoencoder is able to reconstruct the input images. One thing I would add here is that even if an autoencoder can reconstruct images perfectly, it doesn't mean that the encoding it learned is useful. For example, when I wanted similar images to be mapped to similar encodings, the autoencoder that was able to do that better was outputting more blurred reconstructed images in comparison to the autoencoder that wasn't achieving this similarity preservation (but was outputting better reconstructions).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T12:37:34.743Z
Answers.LastActivityDate: 2020-06-24T12:37:34.743Z
Answers.UserDisplayName: nim.py
Body: <p>I am training an autoencoder on (general) image data. </p>
  
  <p>I use binary crossentropy loss function, but it is not very informative when I want to evaluate the performance of my autoencoder. </p>
  
  <p>An obvious performance metric would be pixel-wise MSE, but it has its own downsides, shown on some toy examples in an image from <a href="https://arxiv.org/pdf/2001.03444.pdf" rel="nofollow noreferrer">paper from Pihlgren et al</a>.</p>
  
  <p><a href="https://i.sstatic.net/B0MlB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/B0MlB.png" alt="Examples where MSE is not a useful performance metric"></a></p>
  
  <p>In the same paper, the authors suggest using perceptual loss, but it seems complicated and not well-studied. </p>
  
  <p>I found <a href="https://www.reddit.com/r/MachineLearning/comments/4ptq05/how_to_measure_the_quality_of_the_representations/" rel="nofollow noreferrer">some</a> <a href="https://stats.stackexchange.com/questions/274123/evaluating-an-autoencoder-possible-approaches">other</a> <a href="https://www.quora.com/How-can-I-understand-my-Autoencoder-or-RBM-is-good-without-having-any-labeled-data" rel="nofollow noreferrer">instances</a> of this question, but there doesn't seem to be a concensus.</p>
  
  <p>I understand that it depends on the application, but <strong>I want to know if there are some general guidelines as to which performance metric to use when training autoencoders on image data</strong>.</p>
  
Tags: <autoencoders><image-processing>
Title: How to evaluate the performance of an autoencoder trained on image data?
UserDisplayName: nim.py
ThreadText: # How to evaluate the performance of an autoencoder trained on image data?
  
  (Asked by: username_0 on 2020-05-05)
  
  I am training an autoencoder on (general) image data.
  
  I use binary crossentropy loss function, but it is not very informative when I want to evaluate the performance of my autoencoder.
  
  An obvious performance metric would be pixel-wise MSE, but it has its own downsides, shown on some toy examples in an image from [paper from Pihlgren et al](https://arxiv.org/pdf/2001.03444.pdf).
  
  [![Examples where MSE is not a useful performance metric](https://i.sstatic.net/B0MlB.png)](https://i.sstatic.net/B0MlB.png)
  
  In the same paper, the authors suggest using perceptual loss, but it seems complicated and not well-studied.
  
  I found [some](https://www.reddit.com/r/MachineLearning/comments/4ptq05/how_to_measure_the_quality_of_the_representations/) [other](https://stats.stackexchange.com/questions/274123/evaluating-an-autoencoder-possible-approaches) [instances](https://www.quora.com/How-can-I-understand-my-Autoencoder-or-RBM-is-good-without-having-any-labeled-data) of this question, but there doesn't seem to be a concensus.
  
  I understand that it depends on the application, but **I want to know if there are some general guidelines as to which performance metric to use when training autoencoders on image data**.
  
  **Tags:** autoencoders, image-processing
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-06-24. Score: 1
  
  I will answer my own question to try and provide some insights.
  
  My research supervisor suggested that I should use the [SSIM metric](https://www.cns.nyu.edu/%7Elcv/ssim/) or some other well-known image processing metric (see the book ["Modern Image Quality Assessment" by Wang and Bovik](https://www.morganclaypool.com/doi/abs/10.2200/s00010ed1v01y200508ivm003)) for assessing the visual similarity of an images.
  
  Another way I evaluate the performance of an autoencoder is by simply visually comparing the input and output images taken from the test set. This is by no means very scientific, but it gives a good idea whether an autoencoder is able to reconstruct the input images. One thing I would add here is that even if an autoencoder can reconstruct images perfectly, it doesn't mean that the encoding it learned is useful. For example, when I wanted similar images to be mapped to similar encodings, the autoencoder that was able to do that better was outputting more blurred reconstructed images in comparison to the autoencoder that wasn't achieving this similarity preservation (but was outputting better reconstructions).
  
  
  ---
  
  