Body: <p>Suppose we've got the following formula:</p>
  <p><span class="math-container">$\log p(x;\theta)=\mathbb{E}_{q(z|x;\phi)}[\log p(x,z;\theta)-\log q(z|x;\phi)]+KL(q(z|x;\phi)||p(z|x;\theta))\\ \geq \mathbb{E}_{q(z|x;\phi)}[\log p(x,z;\theta)-\log q(z|x;\phi)]$</span></p>
  <p>We call this <strong>Evidence Lower Bound</strong> (ELBO):</p>
  <p><span class="math-container">$\mathrm{ELBO}=\mathbb{E}_{q(z|x;\phi)}[\log p(x,z;\theta)-\log q(z|x;\phi)]$</span></p>
  <p>We use the method of maximizing ELBO to optimize the parameters <span class="math-container">$\theta$</span> and <span class="math-container">$\phi$</span>:</p>
  <p><span class="math-container">$\max_{\theta,\phi}\mathbb{E}_{\mathcal{D}}\mathbb{E_{q(z|x;\phi)}}[\log p(x,z;\theta)-\log q(z|x;\phi)]$</span></p>
  <p>We can intuitively understand that: for given <span class="math-container">$\theta$</span>, the upper bound of ELBO is <span class="math-container">$\log p(x;\theta)$</span>, so optimizing <span class="math-container">$\phi$</span> allows ELBO to be as close as <span class="math-container">$\log p(x;\theta)$</span> possible; Then, optimizing <span class="math-container">$\theta$</span> can make the data likelihood as large as possible, which achieves the goal of MLE. (just like what shown in Fig)
  <a href="https://i.sstatic.net/bIWtD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bIWtD.png" alt="enter image description here" /></a></p>
  <p>The ELBO mentioned above is also the loss function of VAE. (An algebraic transformation of the ELBO mentioned above: <span class="math-container">$\mathrm{ELBO}=\mathbb{E}_{q(z|x;\phi)}[\log p(x|z;\theta)]-KL(q(z|x;\phi)||p(z;\theta))\\\\$</span>)</p>
  <p>My question is:</p>
  <p>1.When maximizing ELBO, how can I fix <span class="math-container">$\theta$</span> and then <span class="math-container">$\phi$</span>, as mentioned above?</p>
  <p>2.VAE <a href="https://ai.stackexchange.com/questions/24564/how-does-the-implementation-of-the-vaes-objective-function-equate-to-elbo">optimizes them at the same time</a>, there is no precedence. How could I understand the difference?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-30T20:24:09.373Z
Comments.Text: I don't understand your question: "How could I understand the difference?". What difference are you referring to? Can you please also put your actual question in the title? Your question is not how to optimize the ELBO, apparently, if I understand correctly your first question in the body.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-31T02:01:21.397Z
Comments.Text: Thank you for your attention. When applying EM algorithm (Expectation Maximum) to optimize ELBO, we usually fix $\theta$ first and then $\phi$. So I wonder if I can apply to VI(Variational Inference) as well?  The difference i refer to is that one(EM/VI) fixes $\theta$ so that ELBO approaches logp(x), and then optimizes $\theta$ to increase ELBO so that logp(x) increases; The other(VAE) is to maximize ELBO without regard to fixation and sequencing.
Comments.UserDisplayName: Garfield
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-31T06:42:38.670Z
Comments.Text: I have described the problem more clearly(https://ai.stackexchange.com/questions/38537/is-vae-the-same-as-the-e-step-of-the-em-algorithm) and hope to get your help!
Comments.UserDisplayName: Garfield
Tags: <machine-learning><variational-autoencoder><cross-entropy><kl-divergence><evidence-lower-bound>
Title: How to optimize ELBO(VAE's loss function)?
UserDisplayName: Garfield
ThreadText: # How to optimize ELBO(VAE's loss function)?
  
  (Asked by: username_0 on 2022-12-30)
  
  Suppose we've got the following formula:
  
  $\log p(x;\theta)=\mathbb{E}\_{q(z|x;\phi)}[\log p(x,z;\theta)-\log q(z|x;\phi)]+KL(q(z|x;\phi)||p(z|x;\theta))\\ \geq \mathbb{E}\_{q(z|x;\phi)}[\log p(x,z;\theta)-\log q(z|x;\phi)]$
  
  We call this **Evidence Lower Bound** (ELBO):
  
  $\mathrm{ELBO}=\mathbb{E}\_{q(z|x;\phi)}[\log p(x,z;\theta)-\log q(z|x;\phi)]$
  
  We use the method of maximizing ELBO to optimize the parameters $\theta$ and $\phi$:
  
  $\max\_{\theta,\phi}\mathbb{E}\_{\mathcal{D}}\mathbb{E\_{q(z|x;\phi)}}[\log p(x,z;\theta)-\log q(z|x;\phi)]$
  
  We can intuitively understand that: for given $\theta$, the upper bound of ELBO is $\log p(x;\theta)$, so optimizing $\phi$ allows ELBO to be as close as $\log p(x;\theta)$ possible; Then, optimizing $\theta$ can make the data likelihood as large as possible, which achieves the goal of MLE. (just like what shown in Fig)
  [![enter image description here](https://i.sstatic.net/bIWtD.png)](https://i.sstatic.net/bIWtD.png)
  
  The ELBO mentioned above is also the loss function of VAE. (An algebraic transformation of the ELBO mentioned above: $\mathrm{ELBO}=\mathbb{E}\_{q(z|x;\phi)}[\log p(x|z;\theta)]-KL(q(z|x;\phi)||p(z;\theta))\\\\$)
  
  My question is:
  
  1.When maximizing ELBO, how can I fix $\theta$ and then $\phi$, as mentioned above?
  
  2.VAE [optimizes them at the same time](https://ai.stackexchange.com/questions/24564/how-does-the-implementation-of-the-vaes-objective-function-equate-to-elbo), there is no precedence. How could I understand the difference?
  
  **Tags:** machine-learning, variational-autoencoder, cross-entropy, kl-divergence, evidence-lower-bound
  
  > **Comments:**
  > **username_1**: I don't understand your question: "How could I understand the difference?". What difference are you referring to? Can you please also put your actual question in the title? Your question is not how to optimize the ELBO, apparently, if I understand correctly your first question in the body.  
  > **username_0**: Thank you for your attention. When applying EM algorithm (Expectation Maximum) to optimize ELBO, we usually fix $\theta$ first and then $\phi$. So I wonder if I can apply to VI(Variational Inference) as well?  The difference i refer to is that one(EM/VI) fixes $\theta$ so that ELBO approaches logp(x), and then optimizes $\theta$ to increase ELBO so that logp(x) increases; The other(VAE) is to maximize ELBO without regard to fixation and sequencing.  
  > **username_0**: I have described the problem more clearly(https://ai.stackexchange.com/questions/38537/is-vae-the-same-as-the-e-step-of-the-em-algorithm) and hope to get your help!  
  
  ---
  
  ## 0 answers
  
  