Answers.Body: <p>Few more clarifications. While the correct thing to do is draw from the prior, we have no guarantees that the aggregated posterior will cover the prior. Think of the aggregated posterior as the distribution of the latent variables for your dataset (see <a href="https://jmtomczak.github.io/blog/7/7_priors.html" rel="nofollow noreferrer">here</a> for a nice explanation and visualization). Our hope is that this will be like the prior but often in practice we get a mismatch between the prior and the aggregate posterior. In this case sampling from the prior might fail because part of it is not covered by the aggregate posterior. This can be solved in various ways, like learning the prior or computing the aggregated posterior after training.</p>
  <hr>
  <p>Maybe there's a misconception, we are not learning a <code>mu</code> and <code>log_var</code> but a mapping (encoder) from an image to <code>mu</code> and <code>log_var</code>. This is quite different because the <code>mu</code> and <code>log_var</code> are not two fixed vectors for the dataset but are computed separately for each image.</p>
  <p>In similar fashion, the decoder is a learned mapping from the prior distribution <span class="math-container">$N(0,I)$</span> back to the image space.</p>
  <p>Essentially the encoder takes the image as input and spits out the parameters of another gaussian (the posterior). This means that during training the input of the decoder is conditioned upon the image. Let's take MNIST for example. We hope that after the training the encoder has learned to spit out similar <code>mu</code> and <code>log_var</code> for similar digits and that the decoder has learned to decode noise from a posterior to a specific digit.</p>
  <p>For example with a 1-dimensional latent what we hope for is something like this:</p>
  <p>Input digit 0 --&gt; Encoder gives mu 0.1 log_var 0.3<br>
  Input digit 0 --&gt; Encoder gives mu 0.2 log_var 0.2<br>
  Input digit 1 --&gt; Encoder gives mu 1.4 log_var 0.2<br>
  Input digit 1 --&gt; Encoder gives mu 1.5 log_var 0.1<br>
  ...<br>
  Input digit 9 --&gt; Encoder gives mu -4.5 log_var 0.3</p>
  <p><a href="https://becominghuman.ai/using-variational-autoencoder-vae-to-generate-new-images-14328877e88d" rel="nofollow noreferrer">This blogpost</a> has a nice visualization with 2d latents.</p>
  <p>If we didn't have the encoder, we would always draw noise from the same <code>N(0,I)</code> gaussian. This could also work but then we'd need a different training technique like in GANs.</p>
  <p>During test time we many times want to draw a sample from the whole data distribution and for that reason we should use the prior <span class="math-container">$N(0,I)$</span>. If you for some reason want to condition the output to look like a specific sample then you can use the posterior. For example if you only want digits of 1 then you can pass an image of 1 through the encoder and then use the <code>mu</code>, <code>log_var</code> to draw samples.</p>
  <p>So the questions is, do you want a sample from the whole distribution? Then use the prior.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-15T06:58:03.740Z
Answers.Text: I agree that if you always draw noise from N(0, I), its akin to a vanilla GAN which does not model the latent space at all. In a VAE, we are updating the posterior q(z|X) (q=encoder). During training, if you increase the weightage for recon loss, KL-div loss keeps increasing. Now, after training, if you sample from N(0, I), you are not using the learned posterior but the prior which will be _far_ from it. Consequently, your reconstructions look "fuzzy". Only wanting to produce digit 1 is Conditional-VAE, which is conceptually different. So, shouldn't I use posterior instead of N(0, I)?
Answers.UserDisplayName: Arun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-16T08:51:23.037Z
Answers.Text: You don't need to draw from the learned posterior because you're not trying to reconstruct a specific x sample but rather probe the distribution $p(x)$ you have learned. After training the VAE the expected $q(z|x)$ over all $x$ should resemble $p(z)=N(0,I)$. You can see that as $p(z)=\int p(z|x)p(x)$, and if $p(z|x)$ and $q(z|x)$ are close this should be the case for $q(z|x)$ too. In lower dimensions you can "fill" the $N(0,I)$ with a reasonable amount of samples but in higher dimensions this number grows exponentially and your model can cover the whole space so you get blurry images.
Answers.UserDisplayName: sfotiadis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-16T13:26:15.117Z
Answers.Text: many VAE codes have ```total_loss = (alpha * recon_loss) + (beta * kl_loss)```. To get better reconstructions, ```alpha``` is tuned (say, alpha = 300), then, q(z|x) is _far_ from p(z) and sampling from p(z) is suboptimal since the learned q(z|x) gave more weightage to recon_loss and deviated away from p(z). Therefore, in such case(s), sampling from p(z) is not the right way. Correct me if I am wrong?
Answers.UserDisplayName: Arun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-16T15:24:53.083Z
Answers.Text: What you try to optimize in variational inference (hence in VAEs) is the ELBO which is the Evidence Lower Bound (evidence means that comes from the data). The problem is this only a heuristic. Also, due to you model assumptions (ie gaussian latents and decoder) and training simplifications (stochastic estimation of ELBO per batch, not really sampling from the decoder but using moments) it is really unlikely your q(z|x) will be anything close to the real p(z|x).
Answers.UserDisplayName: sfotiadis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-16T15:28:19.190Z
Answers.Text: Now in models like the beta-VAE there is weighting of the ELBO terms. But if you think about it having a weight in front of the reconstruction loss is like saying I like my gaussian decoder to have a smaller variance.  So your models loses some "imagination" (variance). This essentially only affects you decoder assumptions and not how you should sample.
Answers.UserDisplayName: sfotiadis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-13T12:30:42.243Z
Answers.LastActivityDate: 2022-07-05T13:44:49.140Z
Answers.LastEditDate: 2022-07-05T13:44:49.140Z
Answers.UserDisplayName: sfotiadis
Answers.Body: <p>I think method 1 will provide the best output.</p>
  <p>Approximating the empirical distribution of <span class="math-container">$z$</span> should provide decoder inputs in the subset of latent space that the decoder was trained on.</p>
  <p>Sampling from <span class="math-container">$N(0,I)$</span> could undersample or omit some regions of the true distribution, oversample others, and even provide inputs to the decoder that it isn't trained for (and neural networks aren't usually good at extrapolation).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T22:05:23.903Z
Answers.Text: I think so too, but I was told that method #2 is thw _standard way_, to which I am having problems since this is just the prior having no knowledge of our training data. Can you help me prove this?
Answers.UserDisplayName: Arun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-11T00:18:18.807Z
Answers.Text: Yeah, I saw that thread on reddit. I don't know what "the standard way" is, or how one would even find out. Take a poll perhaps?  Neither the original VAE paper nor the $\beta$-VAE paper seem to specify the best way to generate images. The latter does say: "The most informative latent units $z_m$ of $\beta$-VAE have the highest KL divergence from the unit Gaussian prior", confirming at least that the posterior distribution is not $N(0,I)$ and the difference matters. (https://www.deepmind.com/publications/beta-vae-learning-basic-visual-concepts-with-a-constrained-variational-framework)
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-11T00:27:42.110Z
Answers.Text: Perhaps you could investigate this experimentally by finding a latent variable (a single dimension) with a distribution far from standard normal and comparing the results of values for that single latent variable from the two distributions.
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-11T07:06:26.667Z
Answers.Text: Nice find for the deepmind publication! I have been doing this and you can find the code in [here](https://github.com/arjun-majumdar/Synthetic_Data/blob/main/VAE_LSTM-Synthetic_Time_Series_Generation-Household_power_consumption.ipynb) where ```generate_new_data(model, x, n_past, n_features, n_future, random_noise = 1)``` function is used to use 3 different ways to generate data. The question is more of __theoretically justification__.
Answers.UserDisplayName: Arun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-11T10:46:00.997Z
Answers.Text: I think the theoretical justification would go along the lines we've discussed: it's better to sample from the true posterior distribution, which is not $N(0,I)$; sampling from $N(0,I)$ is likely to oversample some regions of the true distribution and undersample others.
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-11T11:26:04.843Z
Answers.Text: I haven't read _Variational inference & deep learning: A new synthesis_ PhD thesis of Kingma, D.P. which should hopefully throw more light on this. But, I agree with you. Say you have two datasets, of course the N(0, 1) does not care about these two different datasets. Whereas, the learned latent space for these 2 would contain relevant information and sampling from this posterior is way better than N(0, 1) prior.
Answers.UserDisplayName: Arun
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-10T14:32:49.797Z
Answers.LastActivityDate: 2022-06-11T11:34:11.430Z
Answers.LastEditDate: 2022-06-11T11:34:11.430Z
Answers.UserDisplayName: Lee Reeves
Body: <p>To generate synthetic dataset using a trained VAE, there is confusion between two approaches:</p>
  <ol>
  <li><p>Use learned latent space: <code>z = mu + (eps * log_var)</code> to generate (theoretically, infinite amounts of) data. Here, we are learning <code>mu</code> and <code>log_var</code> vectors using the data, and, <code>eps</code> is sampled from multivariate, standard, Gaussian distribution.</p>
  </li>
  <li><p>Use multivariate, standard, Gaussian distribution.</p>
  </li>
  </ol>
  <p>I am leaning more towards point 1 since we learn the <code>mu</code> and <code>log_var</code> vectors using our dataset. Whereas, point 2 uses the uninformative prior which contains no particular information about the dataset.</p>
  <p>One of the reasons of VAE is to be able to learn this &quot;unknown&quot; latent space distribution by constraining it to approximate a multivariate, standard, Gaussian distribution, but at the same time, allow it sufficient flexibility to deviate from it too.</p>
  <p>What are your thoughts? I have implemented some VAE, Conditional VAE codes both in TensorFlow 2 and PyTorch which you can refer to <a href="https://github.com/arjun-majumdar/Autoencoders_Experiments" rel="nofollow noreferrer">here</a>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-10T14:09:46.937Z
Comments.Text: Is log_var the log variance of the empirical latent distribution? (I'll denote this $\log \sigma_E^2$)? And if so, shouldn't you calculate $z = \epsilon \sigma_E + \mu$ to give $z$ the distribution $N(\mu, \sigma_E^2$)?
Comments.UserDisplayName: Lee Reeves
Tags: <tensorflow><pytorch><generative-model><variational-autoencoder><latent-variable>
Title: How to generate new data given a trained VAE - sample from the learned latent space or from multivariate Gaussian?
UserDisplayName: Arun
ThreadText: # How to generate new data given a trained VAE - sample from the learned latent space or from multivariate Gaussian?
  
  (Asked by: username_0 on 2022-06-10)
  
  To generate synthetic dataset using a trained VAE, there is confusion between two approaches:
  
  1. Use learned latent space: `z = mu + (eps * log_var)` to generate (theoretically, infinite amounts of) data. Here, we are learning `mu` and `log_var` vectors using the data, and, `eps` is sampled from multivariate, standard, Gaussian distribution.
  2. Use multivariate, standard, Gaussian distribution.
  
  I am leaning more towards point 1 since we learn the `mu` and `log_var` vectors using our dataset. Whereas, point 2 uses the uninformative prior which contains no particular information about the dataset.
  
  One of the reasons of VAE is to be able to learn this "unknown" latent space distribution by constraining it to approximate a multivariate, standard, Gaussian distribution, but at the same time, allow it sufficient flexibility to deviate from it too.
  
  What are your thoughts? I have implemented some VAE, Conditional VAE codes both in TensorFlow 2 and PyTorch which you can refer to [here](https://github.com/arjun-majumdar/Autoencoders_Experiments).
  
  **Tags:** tensorflow, pytorch, generative-model, variational-autoencoder, latent-variable
  > **Comments:**
  > **username_1**: Is log_var the log variance of the empirical latent distribution? (I'll denote this $\log \sigma_E^2$)? And if so, shouldn't you calculate $z = \epsilon \sigma_E + \mu$ to give $z$ the distribution $N(\mu, \sigma_E^2$)?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-06-13. Score: 3
  
  Few more clarifications. While the correct thing to do is draw from the prior, we have no guarantees that the aggregated posterior will cover the prior. Think of the aggregated posterior as the distribution of the latent variables for your dataset (see [here](https://jmtomczak.github.io/blog/7/7_priors.html) for a nice explanation and visualization). Our hope is that this will be like the prior but often in practice we get a mismatch between the prior and the aggregate posterior. In this case sampling from the prior might fail because part of it is not covered by the aggregate posterior. This can be solved in various ways, like learning the prior or computing the aggregated posterior after training.
  
  ---
  
  Maybe there's a misconception, we are not learning a `mu` and `log_var` but a mapping (encoder) from an image to `mu` and `log_var`. This is quite different because the `mu` and `log_var` are not two fixed vectors for the dataset but are computed separately for each image.
  
  In similar fashion, the decoder is a learned mapping from the prior distribution $N(0,I)$ back to the image space.
  
  Essentially the encoder takes the image as input and spits out the parameters of another gaussian (the posterior). This means that during training the input of the decoder is conditioned upon the image. Let's take MNIST for example. We hope that after the training the encoder has learned to spit out similar `mu` and `log_var` for similar digits and that the decoder has learned to decode noise from a posterior to a specific digit.
  
  For example with a 1-dimensional latent what we hope for is something like this:
  
  Input digit 0 --> Encoder gives mu 0.1 log\_var 0.3  
  Input digit 0 --> Encoder gives mu 0.2 log\_var 0.2  
  Input digit 1 --> Encoder gives mu 1.4 log\_var 0.2  
  Input digit 1 --> Encoder gives mu 1.5 log\_var 0.1  
  ...  
  Input digit 9 --> Encoder gives mu -4.5 log\_var 0.3
  
  [This blogpost](https://becominghuman.ai/using-variational-autoencoder-vae-to-generate-new-images-14328877e88d) has a nice visualization with 2d latents.
  
  If we didn't have the encoder, we would always draw noise from the same `N(0,I)` gaussian. This could also work but then we'd need a different training technique like in GANs.
  
  During test time we many times want to draw a sample from the whole data distribution and for that reason we should use the prior $N(0,I)$. If you for some reason want to condition the output to look like a specific sample then you can use the posterior. For example if you only want digits of 1 then you can pass an image of 1 through the encoder and then use the `mu`, `log_var` to draw samples.
  
  So the questions is, do you want a sample from the whole distribution? Then use the prior.
  
  > **Comments:**
  > **username_0**: I agree that if you always draw noise from N(0, I), its akin to a vanilla GAN which does not model the latent space at all. In a VAE, we are updating the posterior q(z|X) (q=encoder). During training, if you increase the weightage for recon loss, KL-div loss keeps increasing. Now, after training, if you sample from N(0, I), you are not using the learned posterior but the prior which will be _far_ from it. Consequently, your reconstructions look "fuzzy". Only wanting to produce digit 1 is Conditional-VAE, which is conceptually different. So, shouldn't I use posterior instead of N(0, I)?  
  > **username_2**: You don't need to draw from the learned posterior because you're not trying to reconstruct a specific x sample but rather probe the distribution $p(x)$ you have learned. After training the VAE the expected $q(z|x)$ over all $x$ should resemble $p(z)=N(0,I)$. You can see that as $p(z)=\int p(z|x)p(x)$, and if $p(z|x)$ and $q(z|x)$ are close this should be the case for $q(z|x)$ too. In lower dimensions you can "fill" the $N(0,I)$ with a reasonable amount of samples but in higher dimensions this number grows exponentially and your model can cover the whole space so you get blurry images.  
  > **username_0**: many VAE codes have ```total_loss = (alpha * recon_loss) + (beta * kl_loss)```. To get better reconstructions, ```alpha``` is tuned (say, alpha = 300), then, q(z|x) is _far_ from p(z) and sampling from p(z) is suboptimal since the learned q(z|x) gave more weightage to recon_loss and deviated away from p(z). Therefore, in such case(s), sampling from p(z) is not the right way. Correct me if I am wrong?  
  > **username_2**: What you try to optimize in variational inference (hence in VAEs) is the ELBO which is the Evidence Lower Bound (evidence means that comes from the data). The problem is this only a heuristic. Also, due to you model assumptions (ie gaussian latents and decoder) and training simplifications (stochastic estimation of ELBO per batch, not really sampling from the decoder but using moments) it is really unlikely your q(z|x) will be anything close to the real p(z|x).  
  > **username_2**: Now in models like the beta-VAE there is weighting of the ELBO terms. But if you think about it having a weight in front of the reconstruction loss is like saying I like my gaussian decoder to have a smaller variance.  So your models loses some "imagination" (variance). This essentially only affects you decoder assumptions and not how you should sample.  
  
  ---
  
  ### Answer by username_1 on 2022-06-10. Score: 0
  
  I think method 1 will provide the best output.
  
  Approximating the empirical distribution of $z$ should provide decoder inputs in the subset of latent space that the decoder was trained on.
  
  Sampling from $N(0,I)$ could undersample or omit some regions of the true distribution, oversample others, and even provide inputs to the decoder that it isn't trained for (and neural networks aren't usually good at extrapolation).
  
  > **Comments:**
  > **username_0**: I think so too, but I was told that method #2 is thw _standard way_, to which I am having problems since this is just the prior having no knowledge of our training data. Can you help me prove this?  
  > **username_1**: Yeah, I saw that thread on reddit. I don't know what "the standard way" is, or how one would even find out. Take a poll perhaps?  Neither the original VAE paper nor the $\beta$-VAE paper seem to specify the best way to generate images. The latter does say: "The most informative latent units $z_m$ of $\beta$-VAE have the highest KL divergence from the unit Gaussian prior", confirming at least that the posterior distribution is not $N(0,I)$ and the difference matters. (https://www.deepmind.com/publications/beta-vae-learning-basic-visual-concepts-with-a-constrained-variational-framework)  
  > **username_1**: Perhaps you could investigate this experimentally by finding a latent variable (a single dimension) with a distribution far from standard normal and comparing the results of values for that single latent variable from the two distributions.  
  > **username_0**: Nice find for the deepmind publication! I have been doing this and you can find the code in [here](https://github.com/arjun-majumdar/Synthetic_Data/blob/main/VAE_LSTM-Synthetic_Time_Series_Generation-Household_power_consumption.ipynb) where ```generate_new_data(model, x, n_past, n_features, n_future, random_noise = 1)``` function is used to use 3 different ways to generate data. The question is more of __theoretically justification__.  
  > **username_1**: I think the theoretical justification would go along the lines we've discussed: it's better to sample from the true posterior distribution, which is not $N(0,I)$; sampling from $N(0,I)$ is likely to oversample some regions of the true distribution and undersample others.  
  > **username_0**: I haven't read _Variational inference & deep learning: A new synthesis_ PhD thesis of Kingma, D.P. which should hopefully throw more light on this. But, I agree with you. Say you have two datasets, of course the N(0, 1) does not care about these two different datasets. Whereas, the learned latent space for these 2 would contain relevant information and sampling from this posterior is way better than N(0, 1) prior.  
  
  ---
  
  