Answers.Body: <p>The variational distribution is the <strong>distribution (or set of distributions) that you use to approximate the distribution you are looking for</strong>. It's often denoted by <span class="math-container">$q$</span>, <span class="math-container">$q_\phi$</span> or <span class="math-container">$q_\phi(z \mid x)$</span>, where <span class="math-container">$\phi$</span> are the variational parameters. For example, in the case of a Gaussian distribution, <span class="math-container">$\phi = \{ \mu, \sigma \}$</span>. In the case of the VAE, <span class="math-container">$q_\phi(z \mid x)$</span> approximates <span class="math-container">$p(z \mid x)$</span> and the encoder outputs <span class="math-container">$\phi$</span>, which can be used to sample <span class="math-container">$z$</span>, a latent vector, which encodes the original input <span class="math-container">$x$</span> (e.g. an image).</p>
  <p>So, the variational distribution is not the latent space or vector. However, you can speak of variational parameters, which would be <span class="math-container">$\phi$</span>, and the latent vector is sampled from the variational distribution.</p>
  <p>The VAEs use a general approach to learn distributions, which is called <strong>variational inference</strong>. I think the term <strong>variational</strong> comes from <strong>calculus of variations</strong>, which studies how to optimize functionals, like the ELBO, which is optimized in the VAE. You can read about it in <a href="https://mbernste.github.io/posts/variational_inference/" rel="nofollow noreferrer">this blog post</a> (I didn't really read it but it looks ok).</p>
  <p><a href="https://github.com/pytorch/examples/blob/main/vae/main.py" rel="nofollow noreferrer">This</a> is the simplest implementation of the VAE that I know of, which I recommend you check out.</p>
  <p>Right now, I don't know anything about diffusion models, so I don't know if they adopt these techniques.</p>
  <p>(I did my master's thesis, a few years ago, on the application of Bayesian neural networks, some of which are based on variational inference, to a specific context)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-22T02:10:53.943Z
Answers.Text: Thank you very much. I get it in the case of VAE.
Answers.UserDisplayName: diffusion stable
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-23T09:32:56.570Z
Answers.Text: @diffusionstable If the diffusion models use variational inference, then the concept of the variational distribution that I describe here should also apply to them. But maybe someone will provide an answer that addresses that specific case.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-22T00:31:36.303Z
Answers.LastActivityDate: 2023-08-22T00:37:46.230Z
Answers.LastEditDate: 2023-08-22T00:37:46.230Z
Answers.UserDisplayName: nbro
Body: <p>What specifically does the term &quot;variational distribution&quot; refer to?<br></p>
  <ul>
  <li>The encoder of Variational autoencoder?<br></li>
  <li>Forward process of denoising diffusion probabilistic models?<br></li>
  <li>Images or latent variables?<br>
  <a href="https://i.sstatic.net/2t3u6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2t3u6.png" alt="enter image description here" /></a></li>
  </ul>
  
Tags: <machine-learning><variational-autoencoder><diffusion-models>
Title: What exactly is meant by variational distribution?
UserDisplayName: diffusion stable
ThreadText: # What exactly is meant by variational distribution?
  
  (Asked by: username_0 on 2023-08-22)
  
  What specifically does the term "variational distribution" refer to?
  
  * The encoder of Variational autoencoder?
  * Forward process of denoising username_0 probabilistic models?
  * Images or latent variables?  
    [![enter image description here](https://i.sstatic.net/2t3u6.png)](https://i.sstatic.net/2t3u6.png)
  
  **Tags:** machine-learning, variational-autoencoder, diffusion-models
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-22. Score: 4
  
  The variational distribution is the **distribution (or set of distributions) that you use to approximate the distribution you are looking for**. It's often denoted by $q$, $q\_\phi$ or $q\_\phi(z \mid x)$, where $\phi$ are the variational parameters. For example, in the case of a Gaussian distribution, $\phi = \{ \mu, \sigma \}$. In the case of the VAE, $q\_\phi(z \mid x)$ approximates $p(z \mid x)$ and the encoder outputs $\phi$, which can be used to sample $z$, a latent vector, which encodes the original input $x$ (e.g. an image).
  
  So, the variational distribution is not the latent space or vector. However, you can speak of variational parameters, which would be $\phi$, and the latent vector is sampled from the variational distribution.
  
  The VAEs use a general approach to learn distributions, which is called **variational inference**. I think the term **variational** comes from **calculus of variations**, which studies how to optimize functionals, like the ELBO, which is optimized in the VAE. You can read about it in [this blog post](https://mbernste.github.io/posts/variational_inference/) (I didn't really read it but it looks ok).
  
  [This](https://github.com/pytorch/examples/blob/main/vae/main.py) is the simplest implementation of the VAE that I know of, which I recommend you check out.
  
  Right now, I don't know anything about username_0 models, so I don't know if they adopt these techniques.
  
  (I did my master's thesis, a few years ago, on the application of Bayesian neural networks, some of which are based on variational inference, to a specific context)
  
  > **Comments:**
  > **username_0**: Thank you very much. I get it in the case of VAE.  
  > **username_1**: @username_0 If the username_0 models use variational inference, then the concept of the variational distribution that I describe here should also apply to them. But maybe someone will provide an answer that addresses that specific case.  
  
  ---
  
  