Answers.Body: <p>You can just insist each training batch is normally distributed—this is the main idea behind <a href="https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial11/NF_image_modeling.html" rel="nofollow noreferrer">normalizing flows</a>, where you try to maximize the likelihood of sampling the encoding assuming it follows a normal distribution.</p>
  <p>People mostly use VAEs because</p>
  <ol>
  <li>VAEs became popular first, which is because</li>
  <li>Normalizing flows require more math/numerical methods, and</li>
  <li>Take more compute to train.</li>
  </ol>
  <p>Unfortunately, (1) has gotten to the point where everyone is using VAEs, simply because everyone else is! Research and compute is slowly making normalizing flows more viable. For example, you can pair them with <a href="https://arxiv.org/pdf/1806.07366.pdf" rel="nofollow noreferrer">neural ODEs</a> for smoother, dynamic flows, but (as of March 2024) you'd likely have to write your own GPU kernels to get an efficient ODE solver.</p>
  <p>A recent success with normalizing flows is <a href="https://arxiv.org/pdf/2312.01479.pdf" rel="nofollow noreferrer">voice cloning</a>: you can flow from an arbitrary voice to the international phonetic alphabet, which removes timbre. Flowing <em>backward</em> lets you clone someone else's voice.</p>
  <p>Now, back to VAEs. You're absolutely correct that you should not expect perfect reconstruction after adding noise. The issue with traditional encoders is they may get perfect reconstruction, but only for your sample of images. For example, perhaps the encoder only needs to signal &quot;this is a horse&quot;, and because there is only one horse in your dataset the decoder can perfectly reconstruct it. The point of the noise term in VAEs is to splat out your distribution so the representation has to actually be about the image. See <a href="https://stats.stackexchange.com/questions/324340/when-should-i-use-a-variational-autoencoder-as-opposed-to-an-autoencoder/421660#421660">this</a> previous StackExchange answer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-29T23:27:50.853Z
Answers.LastActivityDate: 2024-03-29T23:27:50.853Z
Answers.UserDisplayName: programjames
Body: <p>I had come to think of VAEs as having 2 loss terms - A reconstruction loss that  tries to ensure that the input matches the output (just as a regular Auto Encoder), and a KL-Divergence Loss term that tries to ensure each training batch is mapped to a smaller dimensional space where the data being mapped is normally distributed at the latent encoding layer with a std normal distribution of <span class="math-container">$\mu$</span>=0 and <span class="math-container">$\sigma^2$</span>=1. The output of the encoder is a <span class="math-container">$\mu$</span> vector and a <span class="math-container">$\sigma^2$</span> vector.</p>
  <p>But, I can't avoid noticing that we add <span class="math-container">$\mathcal{N}(0,1)$</span> noise to each sample at the latent encoding layer. So, if our data has an <span class="math-container">$\mathcal{N}(0,1)$</span> distribution and we then add exactly this type of noise to each sample in the representation layer, why do we expect successful reconstruction? It seems like we've added a lot of noise, given the region over which our data is distributed.</p>
  <p>Is it just because on 'average' no noise has been added?</p>
  <p>Why not train by just letting the kl-loss term insist that each batch have <span class="math-container">$\mu$</span>=0 and <span class="math-container">$\sigma^2$</span>=1 and only take a <span class="math-container">$\mathcal{N}(0,1)$</span> sample from the hidden layer when we actually do want a new hidden sample?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-29T16:38:46.200Z
Comments.Text: You claimed "*I can't avoid noticing that we add $\mathcal{N}(0,1)$ noise to each sample at the hidden layer*", in what reference you noticed this unit Gaussian noise is added at the hidden layer? Encoder or decoder's hidden layer?
Comments.UserDisplayName: cinch
Tags: <variational-autoencoder>
Title: Why do we add noise when training VAEs, instead of just insisting each training batch is Normally distributed at hidden layer?
UserDisplayName: Steven
ThreadText: # Why do we add noise when training VAEs, instead of just insisting each training batch is Normally distributed at hidden layer?
  
  (Asked by: username_0 on 2024-03-29)
  
  I had come to think of VAEs as having 2 loss terms - A reconstruction loss that tries to ensure that the input matches the output (just as a regular Auto Encoder), and a KL-Divergence Loss term that tries to ensure each training batch is mapped to a smaller dimensional space where the data being mapped is normally distributed at the latent encoding layer with a std normal distribution of $\mu$=0 and $\sigma^2$=1. The output of the encoder is a $\mu$ vector and a $\sigma^2$ vector.
  
  But, I can't avoid noticing that we add $\mathcal{N}(0,1)$ noise to each sample at the latent encoding layer. So, if our data has an $\mathcal{N}(0,1)$ distribution and we then add exactly this type of noise to each sample in the representation layer, why do we expect successful reconstruction? It seems like we've added a lot of noise, given the region over which our data is distributed.
  
  Is it just because on 'average' no noise has been added?
  
  Why not train by just letting the kl-loss term insist that each batch have $\mu$=0 and $\sigma^2$=1 and only take a $\mathcal{N}(0,1)$ sample from the hidden layer when we actually do want a new hidden sample?
  
  **Tags:** variational-autoencoder
  
  > **Comments:**
  > **username_1**: You claimed "*I can't avoid noticing that we add $\mathcal{N}(0,1)$ noise to each sample at the hidden layer*", in what reference you noticed this unit Gaussian noise is added at the hidden layer? Encoder or decoder's hidden layer?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-03-29. Score: 2
  
  You can just insist each training batch is normally distributed—this is the main idea behind [normalizing flows](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial11/NF_image_modeling.html), where you try to maximize the likelihood of sampling the encoding assuming it follows a normal distribution.
  
  People mostly use VAEs because
  
  1. VAEs became popular first, which is because
  2. Normalizing flows require more math/numerical methods, and
  3. Take more compute to train.
  
  Unfortunately, (1) has gotten to the point where everyone is using VAEs, simply because everyone else is! Research and compute is slowly making normalizing flows more viable. For example, you can pair them with [neural ODEs](https://arxiv.org/pdf/1806.07366.pdf) for smoother, dynamic flows, but (as of March 2024) you'd likely have to write your own GPU kernels to get an efficient ODE solver.
  
  A recent success with normalizing flows is [voice cloning](https://arxiv.org/pdf/2312.01479.pdf): you can flow from an arbitrary voice to the international phonetic alphabet, which removes timbre. Flowing *backward* lets you clone someone else's voice.
  
  Now, back to VAEs. You're absolutely correct that you should not expect perfect reconstruction after adding noise. The issue with traditional encoders is they may get perfect reconstruction, but only for your sample of images. For example, perhaps the encoder only needs to signal "this is a horse", and because there is only one horse in your dataset the decoder can perfectly reconstruct it. The point of the noise term in VAEs is to splat out your distribution so the representation has to actually be about the image. See [this](https://stats.stackexchange.com/questions/324340/when-should-i-use-a-variational-autoencoder-as-opposed-to-an-autoencoder/421660#421660) previous StackExchange answer.
  
  
  ---
  
  