Answers.Body: <p>GANs generally produce better photo-realistic images but can be difficult to work with.  Conversely, VAEs are easier to train but don’t usually give the best results.</p>
  <p>I recommend picking VAEs if you don’t have a lot of time to experiment with GANs and photorealism isn’t paramount.</p>
  <p>There are exceptions such as Google’s VQ-VAE 2 which can compete with GANs for image quality and realism.  There is also VAE-GAN and VQ-VAE-GAN.</p>
  <p>As a note, GANs and VAEs are not specifically for images and can be used for other data types/structures.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-06T12:28:34.077Z
Answers.Text: Thanks Brian. I think this answers the question, although I will also change my question a bit as I realised this is not exactly what I'm looking for. Looking to get something a little more fundamental.
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-06T12:25:36.963Z
Answers.LastActivityDate: 2021-01-06T12:25:36.963Z
Answers.UserDisplayName: Brian O'Donnell
Body: <p>Starting from my own understanding, and scoped to the purpose of image generation, I'm well aware of the major architectural differences:</p>
  <ul>
  <li><p>A GAN's generator samples from a relatively low dimensional random variable and produces an image. Then the discriminator takes that image and predicts whether the image belongs to a target distribution or not. Once trained, it can generate a variety of images just by sampling the initial random variable and forwarding through the generator.</p>
  </li>
  <li><p>A VAE's encoder takes an image from a target distribution and compresses it into a low dimensional latent space. Then the decoder's job is to take that latent space representation and reproduce the original image. Once the network is trained, it can generate latent space representations of various images, and interpolate between these before forwarding through the decoder which produces new images.</p>
  </li>
  </ul>
  <p><strong>What I'm more interested is the consequences of said architectural differences.</strong> Why would I choose one approach over the other? And why? (for example, if GANs typically produce better quality images, any ideas why that is so? is it true in all cases or just some?)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-06T11:49:54.090Z
Comments.Text: You're already describing both approaches, and, as far as I know, these descriptions are accurate. So, are you asking that someone tries to tell you differences that you are not able to capture from your description? Is this your first question? I suppose, of course, you're also interested in knowing if you missed something in terms of the high-level description of both the VAE and GAN.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-06T11:50:10.150Z
Comments.Text: The other question "when would you use one over the other" is actually very interesting and useful, I would say, given that they are both generative models, and it's not clear when one would perform better than the other, so when to choose one over the other.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-06T11:55:25.870Z
Comments.Text: @nbro yes I mean the more interesting question. I hoped using "fundamental" in the title would hint that I don't care about the obvious architectural differences. I'll edit to try and make it clearer.
Comments.UserDisplayName: Alexander Soare
Tags: <comparison><applications><generative-adversarial-networks><variational-autoencoder><image-generation>
Title: What are the fundamental differences between VAE and GAN for image generation?
UserDisplayName: Alexander Soare
ThreadText: # What are the fundamental differences between VAE and GAN for image generation?
  
  (Asked by: username_0 on 2021-01-06)
  
  Starting from my own understanding, and scoped to the purpose of image generation, I'm well aware of the major architectural differences:
  
  * A GAN's generator samples from a relatively low dimensional random variable and produces an image. Then the discriminator takes that image and predicts whether the image belongs to a target distribution or not. Once trained, it can generate a variety of images just by sampling the initial random variable and forwarding through the generator.
  * A VAE's encoder takes an image from a target distribution and compresses it into a low dimensional latent space. Then the decoder's job is to take that latent space representation and reproduce the original image. Once the network is trained, it can generate latent space representations of various images, and interpolate between these before forwarding through the decoder which produces new images.
  
  **What I'm more interested is the consequences of said architectural differences.** Why would I choose one approach over the other? And why? (for example, if GANs typically produce better quality images, any ideas why that is so? is it true in all cases or just some?)
  
  **Tags:** comparison, applications, generative-adversarial-networks, variational-autoencoder, image-generation
  
  > **Comments:**
  > **username_1**: You're already describing both approaches, and, as far as I know, these descriptions are accurate. So, are you asking that someone tries to tell you differences that you are not able to capture from your description? Is this your first question? I suppose, of course, you're also interested in knowing if you missed something in terms of the high-level description of both the VAE and GAN.  
  > **username_1**: The other question "when would you use one over the other" is actually very interesting and useful, I would say, given that they are both generative models, and it's not clear when one would perform better than the other, so when to choose one over the other.  
  > **username_0**: @username_1 yes I mean the more interesting question. I hoped using "fundamental" in the title would hint that I don't care about the obvious architectural differences. I'll edit to try and make it clearer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-01-06. Score: 6
  
  GANs generally produce better photo-realistic images but can be difficult to work with. Conversely, VAEs are easier to train but don’t usually give the best results.
  
  I recommend picking VAEs if you don’t have a lot of time to experiment with GANs and photorealism isn’t paramount.
  
  There are exceptions such as Google’s VQ-VAE 2 which can compete with GANs for image quality and realism. There is also VAE-GAN and VQ-VAE-GAN.
  
  As a note, GANs and VAEs are not specifically for images and can be used for other data types/structures.
  
  > **Comments:**
  > **username_0**: Thanks username_2. I think this answers the question, although I will also change my question a bit as I realised this is not exactly what I'm looking for. Looking to get something a little more fundamental.  
  
  ---
  
  