Answers.Body: <p>[Answering my own question after 5 months of studying VAE models]</p>
  
  <p>The point of the MMD-VAE or InfoVAE is not exactly to emphasise on the visual quality of generated samples. It is to preserve greater amount of information through the encoding process. The MMD formulation stems from introducing a mutual coefficient factor into the Evidence Lower BOund (ELBO) loss of VAEs. Refer to the <a href="https://arxiv.org/pdf/1706.02262.pdf)%F0%9F%8E%99%EF%B8%8FPoster" rel="nofollow noreferrer">paper</a> appendices for full derivation. This formulation improves information content in latent space and provides for more accurate approximation of the true posterior - these results have also been empirically proven in the paper.</p>
  
  <p>However, the InfoVAE uses pixel-wise or element-wise reconstruction loss. An element-wise reconstruction loss is likely to lead to some extent of blurriness inrespective of the prior loss term. On Github, several developers have implemented the InfoVAE model and shown their results. Here is a <a href="https://github.com/pren1/keras-MMD-Variational-Autoencoder" rel="nofollow noreferrer">link</a> to one such implementation whose results I could personally verify.</p>
  
  <p>From my own experimentations, I can say that even though InfoVAE may give better reconstructions for some data, there is still considerable blurriness.</p>
  
  <p>Perceptual similarity metrics may be learned or computed as a static function of the input image. With a <a href="https://arxiv.org/pdf/1801.03924.pdf" rel="nofollow noreferrer">learned perceptual loss</a>, VAEs can produce much sharper images. <a href="https://arxiv.org/pdf/1611.05013.pdf" rel="nofollow noreferrer">PixelVAE</a> and <a href="https://arxiv.org/abs/1512.09300" rel="nofollow noreferrer">VAEGAN</a> are well-known models with such implementations. For a static function of the image itself, reconstruction quality will depend on the nature of that function and such a model may not be very useful for all kinds of datasets. Using measures like SSIM, FSIM, we may still end up getting blurred images.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T07:12:33.110Z
Answers.LastActivityDate: 2020-04-26T11:17:53.820Z
Answers.LastEditDate: 2020-04-26T11:17:53.820Z
Answers.UserDisplayName: Ananda
Body: <p>I understand that with vanilla VAEs, there are a few reasons justifying the production of blurred out images. The <a href="https://arxiv.org/pdf/1706.02262.pdf)%F0%9F%8E%99%EF%B8%8FPoster" rel="nofollow noreferrer">InfoVAE</a> paper describes the case when the decoder is flexible enough to ignore the latent attributes and generate an averaged out image that best reduces the reconstruction loss. Thus the blurred image.</p>
  
  <p>How much of the problem of blurring is really mitigated by the <a href="https://arxiv.org/pdf/1706.02262.pdf)%F0%9F%8E%99%EF%B8%8FPoster" rel="nofollow noreferrer">MMD formulation</a> in practical experiments? If someone has experience working with <a href="https://arxiv.org/pdf/1706.02262.pdf)%F0%9F%8E%99%EF%B8%8FPoster" rel="nofollow noreferrer">MMD-VAEs</a>, I'd like to know their opinion on what the reconstruction quality of MMD-VAEs is really like.</p>
  
  <p>Also, does the replacement of the MSE reconstruction loss metric by other perceptual similarity metrics improve generated image quality?</p>
  
Tags: <neural-networks><deep-learning><generative-model><image-generation><variational-autoencoder>
Title: Does MMD-VAE solve the problem of blurred images of vanilla VAEs?
UserDisplayName: Ananda
ThreadText: # Does MMD-VAE solve the problem of blurred images of vanilla VAEs?
  
  (Asked by: username_0 on 2019-11-19)
  
  I understand that with vanilla VAEs, there are a few reasons justifying the production of blurred out images. The [InfoVAE](https://arxiv.org/pdf/1706.02262.pdf)%F0%9F%8E%99%EF%B8%8FPoster) paper describes the case when the decoder is flexible enough to ignore the latent attributes and generate an averaged out image that best reduces the reconstruction loss. Thus the blurred image.
  
  How much of the problem of blurring is really mitigated by the [MMD formulation](https://arxiv.org/pdf/1706.02262.pdf)%F0%9F%8E%99%EF%B8%8FPoster) in practical experiments? If someone has experience working with [MMD-VAEs](https://arxiv.org/pdf/1706.02262.pdf)%F0%9F%8E%99%EF%B8%8FPoster), I'd like to know their opinion on what the reconstruction quality of MMD-VAEs is really like.
  
  Also, does the replacement of the MSE reconstruction loss metric by other perceptual similarity metrics improve generated image quality?
  
  **Tags:** neural-networks, deep-learning, generative-model, image-generation, variational-autoencoder
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-04-26. Score: 2 | Accepted answer
  
  [Answering my own question after 5 months of studying VAE models]
  
  The point of the MMD-VAE or InfoVAE is not exactly to emphasise on the visual quality of generated samples. It is to preserve greater amount of information through the encoding process. The MMD formulation stems from introducing a mutual coefficient factor into the Evidence Lower BOund (ELBO) loss of VAEs. Refer to the [paper](https://arxiv.org/pdf/1706.02262.pdf)%F0%9F%8E%99%EF%B8%8FPoster) appendices for full derivation. This formulation improves information content in latent space and provides for more accurate approximation of the true posterior - these results have also been empirically proven in the paper.
  
  However, the InfoVAE uses pixel-wise or element-wise reconstruction loss. An element-wise reconstruction loss is likely to lead to some extent of blurriness inrespective of the prior loss term. On Github, several developers have implemented the InfoVAE model and shown their results. Here is a [link](https://github.com/pren1/keras-MMD-Variational-Autoencoder) to one such implementation whose results I could personally verify.
  
  From my own experimentations, I can say that even though InfoVAE may give better reconstructions for some data, there is still considerable blurriness.
  
  Perceptual similarity metrics may be learned or computed as a static function of the input image. With a [learned perceptual loss](https://arxiv.org/pdf/1801.03924.pdf), VAEs can produce much sharper images. [PixelVAE](https://arxiv.org/pdf/1611.05013.pdf) and [VAEGAN](https://arxiv.org/abs/1512.09300) are well-known models with such implementations. For a static function of the image itself, reconstruction quality will depend on the nature of that function and such a model may not be very useful for all kinds of datasets. Using measures like SSIM, FSIM, we may still end up getting blurred images.
  
  
  ---
  
  