Answers.Body: <p>The key is: VAE usually use a small latent dimension, the information of input is so hard to pass through this bottleneck, meanwhile it tries to minimize the loss with the batch of input data, you should know the result -- VAE can only have a mean and blurry output.</p>
  
  <p>If you increase the bandwidth of the bottleneck, i.e. the size of latent vector, VAE can get a high reconstruction quality, e.g. <a href="https://github.com/seasonyc/Spatial-Z-VAE" rel="nofollow noreferrer">Spatial-Z-VAE</a> </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-24T08:21:32.033Z
Answers.Text: I don't understand why this answer is correct. GAN also requires a low-dimensional vector to generate images.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-15T06:23:41.407Z
Answers.LastActivityDate: 2019-01-15T06:23:41.407Z
Answers.UserDisplayName: seasonyc
Answers.Body: <p>In essence, Variational Autoencoders learn an &quot;explicit&quot; distribution of the data by trying to fit the data via a multi-dimensional Gaussian/Normal distribution.</p>
  <p>However, Generative Adversarial Networks learn an &quot;implicit&quot; distribution of data meaning that you cannot directly sample them.</p>
  <p>Also, due to the deterministic nature of neural networks, GANs tend to learn a <a href="https://en.wikipedia.org/wiki/Dirac_delta_function" rel="nofollow noreferrer">Dirac Delta function</a>. If you're lucky and the training of the GAN is successful, you can therefore get sharper images, since the model doesn't have to explicitly deal with the noise injected into it due to samplings, hence this could be a simpler learning problem.</p>
  <p>By deterministic, I mean assuming that you have no sampling anywhere in the middle layers of your model and only use the neural network as an input-output mapping function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-16T19:49:05.110Z
Answers.Text: @samutamm, as it is mentioned, VAE learnt the distribution of the data by fitting it into a multi-dimension normal distribution. This results at having the generated samples from VAE models to be blurry because of the independence assumption of the samples given latent variables.
  
  Ref: Page 1 in https://openreview.net/pdf?id=B1ElR4cgg
  
  "...suffer from a well-recognized issue of the maximum likelihood training paradigm when combined with a conditional independence assumption on the output given the latent variables: they tend to distribute probability mass diffusely over the data space.."
Answers.UserDisplayName: Yasmin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-10T09:11:03.560Z
Answers.Text: Can you share some sources for your statement "deterministic nature of neural networks GANs tend to learn a Dirac Delta function". It is quite interesting and I would like to look more about it
Answers.UserDisplayName: Trect
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-27T17:06:45.770Z
Answers.Text: Nice comparison of VAE and GAN. I understand that GAN is trained differently, but what about blurriness of the output of Variational Autoencoder? Why is it so hard to VAE to produce sharper images?
Answers.UserDisplayName: samu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-09T20:20:34.797Z
Answers.LastActivityDate: 2021-12-24T08:22:37.883Z
Answers.LastEditDate: 2021-12-24T08:22:37.883Z
Answers.UserDisplayName: Amir
Answers.Body: <p>The reason is because of L1 (or L2) reconstruction loss used in VAEs.
  As is discussed in <a href="https://arxiv.org/pdf/1611.07004.pdf" rel="nofollow noreferrer">Image-to-Image Translation with Conditional Adversarial Networks</a>: &quot; If we take a naive approach and ask the CNN to minimize the
  Euclidean distance between predicted and ground truth pix-
  els, it will tend to produce blurry results. This is because Euclidean distance is minimized by averaging all plausible outputs, which causes blurring&quot;. Further: &quot;  GANs learn a loss that tries to classify if the output image is real or fake, while simultaneously training a generative model to minimize this loss. Blurry images will not be tolerated since they look obviously fake.&quot;</p>
  <p>For further details read the ablation study in 4.2 of linked paper.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-14T01:47:02.857Z
Answers.LastActivityDate: 2022-09-14T01:49:20.547Z
Answers.LastEditDate: 2022-09-14T01:49:20.547Z
Answers.UserDisplayName: hosh0425
Answers.Body: <p><strong>The reason is simple: because VAE's &amp; GANs are almost opposite in their strengths and weaknesses.</strong></p>
  <p>VAE's (even VQ-VAEs!) always introduce some kind of noise to their encoder's output which has two consequences: it <a href="https://www.analyticsvidhya.com/blog/2023/07/an-overview-of-variational-autoencoders/#:%7E:text=Because%20the%20VAE%27s%20latent%20space,density%20estimation%20and%20text%20generation." rel="nofollow noreferrer">makes the latent space interoperable</a> which <a href="https://ai.stackexchange.com/questions/8879/why-doesnt-vae-suffer-mode-collapse">increases generation diversity</a> (the goal), <strong>but the added noise introduces some error/blur (unintended side-effect)</strong>.</p>
  <p>In contrast GAN's always uses adversaries to detect bad (e.g. blurry) images which has two consequences: <strong>the generated fake images are nearly indistinguishable from real images (the goal)</strong>, <a href="https://www.oreilly.com/library/view/hands-on-generative-adversarial/9781789538205/1133a516-b38d-4271-b510-0ba7077f7292.xhtml#:%7E:text=Mode%20collapse%20and%20mode%20drop%20are%20common%20problems%20in%20GANs,but%20some%20modes%20are%20missing." rel="nofollow noreferrer">the generator tends drop modes (aka reduce diversity)</a> to make fooling the adversary easier (unintended side effect).</p>
  <p>P.S. for VQ-VAEs: The noise in introduced by the vector-quantization &amp; limited codebook size which causes a many-to-one (i.e. noisy) mapping.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-23T18:37:01.013Z
Answers.LastActivityDate: 2024-01-23T19:22:39.120Z
Answers.LastEditDate: 2024-01-23T19:22:39.120Z
Answers.UserDisplayName: profPlum
Answers.Body: <p>Fundamentally speaking, the optimization target of a VAE has two parts: <strong>minimizing reconstruction loss</strong> (similar to an AutoEncoder) and <strong>minimizing KL divergence loss</strong> (ensuring that the latent z follows a normal distribution so we can sample from it). However, these two objectives are <strong>antagonistic</strong>. The former aims to <strong>clearly distinguish different classes in the latent space</strong> to facilitate the final reconstruction, while the latter aims to <strong>mix different classes in the latent space</strong> to conform to a standard normal distribution. Therefore, after optimization, the images generated by a VAE are unlikely to be very clear and sharp.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-18T12:51:51.843Z
Answers.LastActivityDate: 2024-06-18T13:20:47.873Z
Answers.LastEditDate: 2024-06-18T13:20:47.873Z
Answers.UserDisplayName: hyukshuai lee
Body: <p>I observed in several papers that the variational autoencoder's output is blurred, while GANs output is crisp and has sharp edges.</p>
  <p>Can someone please give some intuition why that is the case? I did think a lot but couldn't find any logic.</p>
  
Tags: <comparison><generative-adversarial-networks><autoencoders><generative-model><variational-autoencoder>
Title: Why is the variational auto-encoder's output blurred, while GANs output is crisp and has sharp edges?
UserDisplayName: Trect
ThreadText: # Why is the variational auto-encoder's output blurred, while GANs output is crisp and has sharp edges?
  
  (Asked by: username_0 on 2018-11-09)
  
  I observed in several papers that the variational autoencoder's output is blurred, while GANs output is crisp and has sharp edges.
  
  Can someone please give some intuition why that is the case? I did think a lot but couldn't find any logic.
  
  **Tags:** comparison, generative-adversarial-networks, autoencoders, generative-model, variational-autoencoder
  
  ---
  
  ## 5 answers
  
  ### Answer by username_1 on 2019-01-15. Score: 4
  
  The key is: VAE usually use a small latent dimension, the information of input is so hard to pass through this bottleneck, meanwhile it tries to minimize the loss with the batch of input data, you should know the result -- VAE can only have a mean and blurry output.
  
  If you increase the bandwidth of the bottleneck, i.e. the size of latent vector, VAE can get a high reconstruction quality, e.g. [Spatial-Z-VAE](https://github.com/seasonyc/Spatial-Z-VAE)
  
  > **Comments:**
  > **username_2**: I don't understand why this answer is correct. GAN also requires a low-dimensional vector to generate images.  
  
  ---
  
  ### Answer by username_3 on 2018-11-09. Score: 2
  
  In essence, Variational Autoencoders learn an "explicit" distribution of the data by trying to fit the data via a multi-dimensional Gaussian/Normal distribution.
  
  However, Generative Adversarial Networks learn an "implicit" distribution of data meaning that you cannot directly sample them.
  
  Also, due to the deterministic nature of neural networks, GANs tend to learn a [Dirac Delta function](https://en.wikipedia.org/wiki/Dirac_delta_function). If you're lucky and the training of the GAN is successful, you can therefore get sharper images, since the model doesn't have to explicitly deal with the noise injected into it due to samplings, hence this could be a simpler learning problem.
  
  By deterministic, I mean assuming that you have no sampling anywhere in the middle layers of your model and only use the neural network as an input-output mapping function.
  
  > **Comments:**
  > **username_0**: Can you share some sources for your statement "deterministic nature of neural networks GANs tend to learn a Dirac Delta function". It is quite interesting and I would like to look more about it  
  > **username_5**: Nice comparison of VAE and GAN. I understand that GAN is trained differently, but what about blurriness of the output of Variational Autoencoder? Why is it so hard to VAE to produce sharper images?  
  > **username_4**: @samutamm, as it is mentioned, VAE learnt the distribution of the data by fitting it into a multi-dimension normal distribution. This results at having the generated samples from VAE models to be blurry because of the independence assumption of the samples given latent variables.  Ref: Page 1 in https://openreview.net/pdf?id=B1ElR4cgg  "...suffer from a well-recognized issue of the maximum likelihood training paradigm when combined with a conditional independence assumption on the output given the latent variables: they tend to distribute probability mass diffusely over the data space.."  
  
  ---
  
  ### Answer by username_6 on 2022-09-14. Score: 1
  
  The reason is because of L1 (or L2) reconstruction loss used in VAEs.
  As is discussed in [Image-to-Image Translation with Conditional Adversarial Networks](https://arxiv.org/pdf/1611.07004.pdf): " If we take a naive approach and ask the CNN to minimize the
  Euclidean distance between predicted and ground truth pix-
  els, it will tend to produce blurry results. This is because Euclidean distance is minimized by averaging all plausible outputs, which causes blurring". Further: " GANs learn a loss that tries to classify if the output image is real or fake, while simultaneously training a generative model to minimize this loss. Blurry images will not be tolerated since they look obviously fake."
  
  For further details read the ablation study in 4.2 of linked paper.
  
  
  ---
  
  ### Answer by username_7 on 2024-01-23. Score: 0
  
  **The reason is simple: because VAE's & GANs are almost opposite in their strengths and weaknesses.**
  
  VAE's (even VQ-VAEs!) always introduce some kind of noise to their encoder's output which has two consequences: it [makes the latent space interoperable](https://www.analyticsvidhya.com/blog/2023/07/an-overview-of-variational-autoencoders/#:%7E:text=Because%20the%20VAE%27s%20latent%20space,density%20estimation%20and%20text%20generation.) which [increases generation diversity](https://ai.stackexchange.com/questions/8879/why-doesnt-vae-suffer-mode-collapse) (the goal), **but the added noise introduces some error/blur (unintended side-effect)**.
  
  In contrast GAN's always uses adversaries to detect bad (e.g. blurry) images which has two consequences: **the generated fake images are nearly indistinguishable from real images (the goal)**, [the generator tends drop modes (aka reduce diversity)](https://www.oreilly.com/library/view/hands-on-generative-adversarial/9781789538205/1133a516-b38d-4271-b510-0ba7077f7292.xhtml#:%7E:text=Mode%20collapse%20and%20mode%20drop%20are%20common%20problems%20in%20GANs,but%20some%20modes%20are%20missing.) to make fooling the adversary easier (unintended side effect).
  
  P.S. for VQ-VAEs: The noise in introduced by the vector-quantization & limited codebook size which causes a many-to-one (i.e. noisy) mapping.
  
  
  ---
  
  ### Answer by username_8 on 2024-06-18. Score: 0
  
  Fundamentally speaking, the optimization target of a VAE has two parts: **minimizing reconstruction loss** (similar to an AutoEncoder) and **minimizing KL divergence loss** (ensuring that the latent z follows a normal distribution so we can sample from it). However, these two objectives are **antagonistic**. The former aims to **clearly distinguish different classes in the latent space** to facilitate the final reconstruction, while the latter aims to **mix different classes in the latent space** to conform to a standard normal distribution. Therefore, after optimization, the images generated by a VAE are unlikely to be very clear and sharp.
  
  
  ---
  
  