Answers.Body: <p>The prior <span class="math-container">$p(z)$</span> is assumed as part of the problem formulation. A typical case is where <span class="math-container">$z$</span> is a vector of iid normal random variables. The ELBO involves a regularization term which encourages <span class="math-container">$q(z \, | \, x)$</span> to have a similar distribution to <span class="math-container">$p(z)$</span> (the way you've written it, that's the KL term). Thus <span class="math-container">$q(z \, | \, x)$</span> will end up having a similar shape to <span class="math-container">$p(z)$</span>. For example, again assuming <span class="math-container">$z$</span> is a vector of iid normals, if you plot samples of <span class="math-container">$z$</span> drawn from <span class="math-container">$q(z \, | \, x)$</span> you will find it has a roughly spherical shape. <a href="https://www.kaggle.com/rvislaywade/visualizing-mnist-using-a-variational-autoencoder" rel="nofollow noreferrer">If you scroll down to the [16] code block and look at the figure</a> you'll see what I mean. The figure is plotting samples of <span class="math-container">$z$</span>, colored according to what <span class="math-container">$x$</span> is (MNIST example). This is just some random figure I found, and I don't endorse this code, but the image is what you'd expect to see.</p>
  <p>The way we end up with a distribution <span class="math-container">$p(x, z)$</span> is by using the prior. We sample <span class="math-container">$z$</span> according to <span class="math-container">$p(z)$</span>; we've trained the decoder <span class="math-container">$p(x \, | \, z)$</span>, and by definition <span class="math-container">$p(x, z) = p(x \, | \, z) p(z)$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-23T02:12:53.253Z
Answers.Text: Thanks, I will think about what you said..
Answers.UserDisplayName: a12345
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-23T02:13:14.047Z
Answers.Text: Also, I moved my 2nd question to a separate post here https://ai.stackexchange.com/questions/32161/how-does-the-vae-learn-px-z
Answers.UserDisplayName: a12345
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-22T16:07:58.137Z
Answers.LastActivityDate: 2021-10-22T16:07:58.137Z
Answers.UserDisplayName: Taw
Body: <p>I know the encoder is variational posterior <span class="math-container">$q_{\phi}(\mathbf{z} \mid \mathbf{x})$</span>.</p>
  <p>I also know that the decoder represents the likelihood:
  <span class="math-container">$p_{\theta}(\mathbf{x} \mid \mathbf{z})$</span>.</p>
  <p>My question is about the prior <span class="math-container">$\mathrm{p}(\mathbf{z})$</span>.</p>
  <p>I know ELBO can be written as:</p>
  <p><span class="math-container">$$E_{q_{\phi}(\mathbf{z} \mid \mathbf{x})}[\log (p_{\theta}(\mathbf{x} \mid \mathbf{z}))]-\mathrm{D}_{\mathrm{KL}}( q_{\phi}(\mathbf{z} \mid \mathbf{x}) \| \mathrm{p}(\mathbf{z})) \leq \log (p_{\theta}( \mathbf{x}))$$</span></p>
  <p>And for the VAE, the variational posterior is</p>
  <p><span class="math-container">$$ q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x}^{(i)})= \mathcal{N}( \boldsymbol{\mu}^{(i)}, \boldsymbol{\sigma}^{2(i)} \mathbf{I}),$$</span></p>
  <p>and prior is</p>
  <p><span class="math-container">$$ \mathrm{p}(\mathbf{z})=\mathcal{N}( \boldsymbol{0}, \mathbf{I}).$$</span></p>
  <p>So</p>
  <p><span class="math-container">$$\mathrm{D}_{\mathrm{KL}}\left(\mathrm{q}_{\Phi}(\mathbf{z} \mid \mathbf{x}) \| p_{z}(\mathbf{z})\right)=\frac{1}{2} \sum_{j=1}^{J}\left(1+\log \left(\sigma_{j}^{2}\right)-\sigma_{j}^{2}-\mu_{j}^{2}\right)$$</span></p>
  <p>That's one way I know the prior plays a role, in helping determine part of the loss function.</p>
  <p><em>Is there any other role that the prior plays for the VAE?</em></p>
  
Tags: <generative-model><variational-autoencoder>
Title: What are the roles of the prior $\mathrm{p}(\mathbf{z})$ in a VAE?
UserDisplayName: a12345
ThreadText: # What are the roles of the prior $\mathrm{p}(\mathbf{z})$ in a VAE?
  
  (Asked by: username_0 on 2021-10-19)
  
  I know the encoder is variational posterior $q\_{\phi}(\mathbf{z} \mid \mathbf{x})$.
  
  I also know that the decoder represents the likelihood:
  $p\_{\theta}(\mathbf{x} \mid \mathbf{z})$.
  
  My question is about the prior $\mathrm{p}(\mathbf{z})$.
  
  I know ELBO can be written as:
  
  $$E\_{q\_{\phi}(\mathbf{z} \mid \mathbf{x})}[\log (p\_{\theta}(\mathbf{x} \mid \mathbf{z}))]-\mathrm{D}\_{\mathrm{KL}}( q\_{\phi}(\mathbf{z} \mid \mathbf{x}) \| \mathrm{p}(\mathbf{z})) \leq \log (p\_{\theta}( \mathbf{x}))$$
  
  And for the VAE, the variational posterior is
  
  $$ q\_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x}^{(i)})= \mathcal{N}( \boldsymbol{\mu}^{(i)}, \boldsymbol{\sigma}^{2(i)} \mathbf{I}),$$
  
  and prior is
  
  $$ \mathrm{p}(\mathbf{z})=\mathcal{N}( \boldsymbol{0}, \mathbf{I}).$$
  
  So
  
  $$\mathrm{D}\_{\mathrm{KL}}\left(\mathrm{q}\_{\Phi}(\mathbf{z} \mid \mathbf{x}) \| p\_{z}(\mathbf{z})\right)=\frac{1}{2} \sum\_{j=1}^{J}\left(1+\log \left(\sigma\_{j}^{2}\right)-\sigma\_{j}^{2}-\mu\_{j}^{2}\right)$$
  
  That's one way I know the prior plays a role, in helping determine part of the loss function.
  
  *Is there any other role that the prior plays for the VAE?*
  
  **Tags:** generative-model, variational-autoencoder
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-10-22. Score: 2 | Accepted answer
  
  The prior $p(z)$ is assumed as part of the problem formulation. A typical case is where $z$ is a vector of iid normal random variables. The ELBO involves a regularization term which encourages $q(z \, | \, x)$ to have a similar distribution to $p(z)$ (the way you've written it, that's the KL term). Thus $q(z \, | \, x)$ will end up having a similar shape to $p(z)$. For example, again assuming $z$ is a vector of iid normals, if you plot samples of $z$ drawn from $q(z \, | \, x)$ you will find it has a roughly spherical shape. [If you scroll down to the [16] code block and look at the figure](https://www.kaggle.com/rvislaywade/visualizing-mnist-using-a-variational-autoencoder) you'll see what I mean. The figure is plotting samples of $z$, colored according to what $x$ is (MNIST example). This is just some random figure I found, and I don't endorse this code, but the image is what you'd expect to see.
  
  The way we end up with a distribution $p(x, z)$ is by using the prior. We sample $z$ according to $p(z)$; we've trained the decoder $p(x \, | \, z)$, and by definition $p(x, z) = p(x \, | \, z) p(z)$.
  
  > **Comments:**
  > **username_0**: Thanks, I will think about what you said..  
  > **username_0**: Also, I moved my 2nd question to a separate post here https://ai.stackexchange.com/questions/32161/how-does-the-vae-learn-px-z  
  
  ---
  
  