Answers.Body: <p>The VAE models the following directed graphical model (figure 1 from the original VAE paper)</p>
  <p><a href="https://i.sstatic.net/BLCT7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/BLCT7.png" alt="enter image description here" /></a></p>
  <p>So, you have 2 sets of parameters, <span class="math-container">$\boldsymbol{\phi}$</span> and <span class="math-container">$\boldsymbol{\theta}$</span>, and 2 random variables, <span class="math-container">$\mathbf{z}$</span> (latent r.v.) and <span class="math-container">$\mathbf{x}$</span>.</p>
  <p>How can you view this graphical model (in figure 1 above) as a generative model?</p>
  <ol>
  <li><p>First, a sample <span class="math-container">$\mathbf{z}^{(i)}$</span> is generated from the (variational) probability distribution <span class="math-container">$q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})$</span></p>
  </li>
  <li><p>Then, a sample <span class="math-container">$\mathbf{x}^{(i)}$</span> is generated from <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z}^{(i)})$</span></p>
  </li>
  </ol>
  <p>Now, more concretely, let's assume that we have a dataset <span class="math-container">$\mathbf{X}=\left\{\mathbf{x}^{(i)}\right\}_{i=1}^{N}$</span> of <span class="math-container">$N$</span> i.i.d. samples of the random variable <span class="math-container">$\mathbf{x}$</span>. So, each of these <span class="math-container">$\mathbf{x}^{(i)}$</span> has been generated as follows</p>
  <ol>
  <li><p>a sample <span class="math-container">$\mathbf{z}^{(i)}$</span> is generated from some prior distribution <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{z})$</span></p>
  </li>
  <li><p>a sample <span class="math-container">$\mathbf{x}^{(i)}$</span> is generated from some conditional distribution <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z}^{(i)})$</span>.</p>
  </li>
  </ol>
  <p>Now, in the VAE, the encoder represents <span class="math-container">$q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})$</span>, while the decoder represents <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$</span>. If you want to train a VAE, you also need to make an assumption about <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{z})$</span>, for example, you can assume <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{z}) = \mathcal{N}(\mathbf{z} ; \mathbf{0}, \mathbf{I})$</span>. Once trained, you use the variational distribution (encoder) as the prior from which you sample <span class="math-container">$\mathbf{z}^{(i)}$</span> (although we train the VAE as if the variational distribution is an approximation of the true/unknown/intractable posterior given a usually fixed prior and the likelihood/decoder), in order to sample <span class="math-container">$\mathbf{x}^{(i)}$</span>. This is not wrong. In fact, this is just how you usually do Bayesian statistics. You have a prior and a likelihood (and maybe a marginal), then you compute the posterior, which then becomes the new prior. So, if you had more data, you could learn a new variational distribution <span class="math-container">$q'_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})$</span> by assuming that your new prior is  <span class="math-container">$q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})$</span>.</p>
  <p>If you keep in mind the following equation</p>
  <p><span class="math-container">$$p_{\boldsymbol{\theta}}(\mathbf{x}, \mathbf{z})=p_{\boldsymbol{\theta}}(\mathbf{z}) p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$$</span></p>
  <p>it should remind you why the VAE can be viewed as a <a href="https://en.wikipedia.org/wiki/Generative_model" rel="nofollow noreferrer">generative model</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-29T18:38:33.357Z
Answers.Text: Thanks, this is a great explanation. I was wondering if we considered the variational posterior as a prior, but wasn't sure since it wasn't stated anywhere explicitly, so now that is cleared up for me.
Answers.UserDisplayName: a12345
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-29T14:34:37.670Z
Answers.LastActivityDate: 2021-10-29T14:34:37.670Z
Answers.UserDisplayName: nbro
Body: <p>I found the following paragraph from <a href="https://arxiv.org/abs/1906.02691" rel="nofollow noreferrer">An Introduction to
  Variational Autoencoders</a> sounds relevant, but I am not fully understanding it.</p>
  <blockquote>
  <p>A VAE learns stochastic mappings between an observed <span class="math-container">$\mathbf{x}$</span>-space, whose empirical distribution <span class="math-container">$q_{\mathcal{D}}(\mathbf{x})$</span> is typically complicated, and a latent <span class="math-container">$\mathbf{z}$</span>-space, whose distribution can be relatively simple (such as spherical, as in this figure). The generative model learns a joint distribution <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{x}, \mathbf{z})$</span> that is often (but not always) factorized as <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{x}, \mathbf{z})=p_{\boldsymbol{\theta}}(\mathbf{z}) p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$</span>, with a prior distribution over latent space <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{z})$</span>, and a stochastic decoder <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$</span>. The stochastic encoder <span class="math-container">$q_{\phi}(\mathbf{z} \mid \mathbf{x})$</span>, also called inference model, approximates the true but intractable posterior <span class="math-container">$p_{\theta}(\mathbf{z} \mid \mathbf{x})$</span> of the generative model.</p>
  </blockquote>
  <p>How is it that the generative model learns a joint distribution <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{x}, \mathbf{z})$</span> in the case of the VAE? I know that learning the weights of the decoder is learning <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$</span></p>
  
Tags: <papers><generative-model><variational-autoencoder><probability-distribution><variational-inference>
Title: How does the VAE learn a joint distribution?
UserDisplayName: a12345
ThreadText: # How does the VAE learn a joint distribution?
  
  (Asked by: username_0 on 2021-10-23)
  
  I found the following paragraph from [An Introduction to
  Variational Autoencoders](https://arxiv.org/abs/1906.02691) sounds relevant, but I am not fully understanding it.
  
  > A VAE learns stochastic mappings between an observed $\mathbf{x}$-space, whose empirical distribution $q\_{\mathcal{D}}(\mathbf{x})$ is typically complicated, and a latent $\mathbf{z}$-space, whose distribution can be relatively simple (such as spherical, as in this figure). The generative model learns a joint distribution $p\_{\boldsymbol{\theta}}(\mathbf{x}, \mathbf{z})$ that is often (but not always) factorized as $p\_{\boldsymbol{\theta}}(\mathbf{x}, \mathbf{z})=p\_{\boldsymbol{\theta}}(\mathbf{z}) p\_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$, with a prior distribution over latent space $p\_{\boldsymbol{\theta}}(\mathbf{z})$, and a stochastic decoder $p\_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$. The stochastic encoder $q\_{\phi}(\mathbf{z} \mid \mathbf{x})$, also called inference model, approximates the true but intractable posterior $p\_{\theta}(\mathbf{z} \mid \mathbf{x})$ of the generative model.
  
  How is it that the generative model learns a joint distribution $p\_{\boldsymbol{\theta}}(\mathbf{x}, \mathbf{z})$ in the case of the VAE? I know that learning the weights of the decoder is learning $p\_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$
  
  **Tags:** papers, generative-model, variational-autoencoder, probability-distribution, variational-inference
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-10-29. Score: 2 | Accepted answer
  
  The VAE models the following directed graphical model (figure 1 from the original VAE paper)
  
  [![enter image description here](https://i.sstatic.net/BLCT7.png)](https://i.sstatic.net/BLCT7.png)
  
  So, you have 2 sets of parameters, $\boldsymbol{\phi}$ and $\boldsymbol{\theta}$, and 2 random variables, $\mathbf{z}$ (latent r.v.) and $\mathbf{x}$.
  
  How can you view this graphical model (in figure 1 above) as a generative model?
  
  1. First, a sample $\mathbf{z}^{(i)}$ is generated from the (variational) probability distribution $q\_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})$
  2. Then, a sample $\mathbf{x}^{(i)}$ is generated from $p\_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z}^{(i)})$
  
  Now, more concretely, let's assume that we have a dataset $\mathbf{X}=\left\{\mathbf{x}^{(i)}\right\}\_{i=1}^{N}$ of $N$ i.i.d. samples of the random variable $\mathbf{x}$. So, each of these $\mathbf{x}^{(i)}$ has been generated as follows
  
  1. a sample $\mathbf{z}^{(i)}$ is generated from some prior distribution $p\_{\boldsymbol{\theta}}(\mathbf{z})$
  2. a sample $\mathbf{x}^{(i)}$ is generated from some conditional distribution $p\_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z}^{(i)})$.
  
  Now, in the VAE, the encoder represents $q\_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})$, while the decoder represents $p\_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$. If you want to train a VAE, you also need to make an assumption about $p\_{\boldsymbol{\theta}}(\mathbf{z})$, for example, you can assume $p\_{\boldsymbol{\theta}}(\mathbf{z}) = \mathcal{N}(\mathbf{z} ; \mathbf{0}, \mathbf{I})$. Once trained, you use the variational distribution (encoder) as the prior from which you sample $\mathbf{z}^{(i)}$ (although we train the VAE as if the variational distribution is an approximation of the true/unknown/intractable posterior given a usually fixed prior and the likelihood/decoder), in order to sample $\mathbf{x}^{(i)}$. This is not wrong. In fact, this is just how you usually do Bayesian statistics. You have a prior and a likelihood (and maybe a marginal), then you compute the posterior, which then becomes the new prior. So, if you had more data, you could learn a new variational distribution $q'\_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})$ by assuming that your new prior is $q\_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})$.
  
  If you keep in mind the following equation
  
  $$p\_{\boldsymbol{\theta}}(\mathbf{x}, \mathbf{z})=p\_{\boldsymbol{\theta}}(\mathbf{z}) p\_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$$
  
  it should remind you why the VAE can be viewed as a [generative model](https://en.wikipedia.org/wiki/Generative_model).
  
  > **Comments:**
  > **username_0**: Thanks, this is a great explanation. I was wondering if we considered the variational posterior as a prior, but wasn't sure since it wasn't stated anywhere explicitly, so now that is cleared up for me.  
  
  ---
  
  