Answers.Body: <p><em>I will try to answer your questions directly (but I guess I won't be able to), otherwise, this can become quite confusing, given the inconsistencies that can be found across different sources.</em></p>
  <blockquote>
  <p>In <span class="math-container">$logp_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$</span> why is <span class="math-container">$\theta$</span> and param for <span class="math-container">$p$</span> and <span class="math-container">$q$</span>?</p>
  </blockquote>
  <p>In a few words, your equation is wrong because it uses the letters <span class="math-container">$\phi$</span> and <span class="math-container">$\theta$</span> inconsistently.</p>
  <p>If you look more carefully at the <strong>right-hand side</strong> of your equation, you will notice that <span class="math-container">$q_{\theta}$</span> has different parameters, i.e. <span class="math-container">$\theta$</span>, than <span class="math-container">$p_{\phi}$</span>, which has parameters <span class="math-container">$\phi$</span>, so <span class="math-container">$p$</span> and <span class="math-container">$q$</span> have different parameters, and this should be the case, because they are represented by different neural networks in the case of the VAE. <strong>However</strong>, the <strong>left-hand side</strong> uses <span class="math-container">$\theta$</span> as the parameters of <span class="math-container">$p$</span> (while the right-hand side uses <span class="math-container">$\phi$</span> to index <span class="math-container">$p$</span>), so this should already suggest that <strong>the equation is not correct</strong> (as you correctly thought).</p>
  <p>In the case of the VAE, <span class="math-container">$\phi$</span> <strong>usually</strong> represents the parameters (or weights) of the encoder neural network (NN), while <span class="math-container">$\theta$</span> usually represents the parameters of the decoder NN (or vice-versa, but you should just be consistent, which is often not the case in your equation). In fact, in <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">the VAE paper</a>, in <strong>equation 3</strong>, the authors use <span class="math-container">$\phi$</span> to represent the parameters of the encoder <span class="math-container">$q$</span>, while <span class="math-container">$\theta$</span> is used to denote the parameters of the decoder <span class="math-container">$p$</span>.</p>
  <p>So, if you follow the notation in the VAE paper, the ELBO can actually be written something like</p>
  <p><span class="math-container">\begin{align}
  \mathcal{L}(\phi,\theta; \mathbf{x}) 
  &amp;= 
  \mathbb{E}_{\tilde{z} \sim q_{\phi}(\mathbf{z} \mid \mathbf{x})} \left[ \log p_{\theta} (\mathbf{x} \mid \mathbf{z}) \right] - \operatorname{KL} \left(q_{\phi}(\mathbf{z} \mid \mathbf{x}) \| p_{\theta}(\mathbf{z}) \right) \tag{1} \label{1}
  \end{align}</span></p>
  <p>The ELBO loss <span class="math-container">$\mathcal{L}(\phi,\theta; \mathbf{x})$</span> has both parameters (of the encoder and decoder), which will be optimized jointly. Note that I have ignored the indices in the observations <span class="math-container">$\mathbf{x}$</span> (for simplicity), while, in <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">the VAE paper</a>, they are present. Furthermore, note that, both in \ref{1} and in <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">the VAE paper</a>, we use bold letters (because these objects are usually vectors), i.e. <span class="math-container">$\mathbf{x}$</span> and <span class="math-container">$\mathbf{z}$</span>, rather than <span class="math-container">$x$</span> and <span class="math-container">$z$</span> (like in your equation).</p>
  <p>Note also that, even though <span class="math-container">$p_{\theta}(\mathbf{z})$</span> is indexed by <span class="math-container">$\theta$</span>, in reality, this may be an un-parametrized distribution (e.g. a Gaussian with mean <span class="math-container">$0$</span> and variance <span class="math-container">$1$</span>), i.e. not a family of distributions. The use of the index <span class="math-container">$\theta$</span> in <span class="math-container">$p_{\theta}(\mathbf{z})$</span> comes from the (implicit) assumption that both <span class="math-container">$p_{\theta}(\mathbf{z})$</span> and <span class="math-container">$p_{\theta} (\mathbf{x} \mid \mathbf{z})$</span> come from the same family of distributions (e.g. a family of Gaussians). In fact, if you consider the family of all Gaussian distributions, then <span class="math-container">$p_{\theta}(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I})$</span> also belongs to that family. But <span class="math-container">$\theta$</span> and <span class="math-container">$\phi$</span> are also used to denote the parameters (or weights) of the networks, so this becomes understanbly confusing. (To understand equation 10 of the VAE paper, see <a href="https://ai.stackexchange.com/a/26408/2444">this answer</a>).</p>
  <blockquote>
  <p>Why does <span class="math-container">$p(x^1,...,x^N)$</span> and <span class="math-container">$q(z|x^i)$</span> have the same parameter <span class="math-container">$\theta?$</span></p>
  </blockquote>
  <p>This is <strong>wrong</strong>, in fact. If you look at equation 1 of <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">the VAE paper</a>, they use <span class="math-container">$\theta$</span> to denote the parameters of <span class="math-container">$p(\mathbf{x})$</span>, i.e. <span class="math-container">$p_{\theta}(\mathbf{x})$</span>, while the parameters of the encoder are <span class="math-container">$\phi$</span>, i.e. <span class="math-container">$q_{\phi}(\mathbf{z} \mid \mathbf{x}$</span>).</p>
  <blockquote>
  <p>Cause <span class="math-container">$p$</span> is just the probability of the observed data and <span class="math-container">$q$</span> is the approximation of the posterior so shouldn't they be different distributions and their parameters different?</p>
  </blockquote>
  <p>Yes.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-07T23:44:10.423Z
Answers.LastActivityDate: 2022-06-04T14:32:56.623Z
Answers.LastEditDate: 2022-06-04T14:32:56.623Z
Answers.UserDisplayName: nbro
Body: <p>In <span class="math-container">$$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i),$$</span> why does <span class="math-container">$p(x^1,...,x^N)$</span> and <span class="math-container">$q(z|x^i)$</span> have the same parameter <span class="math-container">$\theta?$</span></p>
  <p>Given that <span class="math-container">$p$</span> is just the probability of the observed data and <span class="math-container">$q$</span> is the approximation of the posterior, shouldn't they be different distributions and thus their parameters different?</p>
  
Tags: <variational-autoencoder><latent-variable><evidence-lower-bound>
Title: In this VAE formula, why do $p$ and $q$ have the same parameters?
UserDisplayName: user8714896
ThreadText: # In this VAE formula, why do $p$ and $q$ have the same parameters?
  
  (Asked by: username_0 on 2020-02-07)
  
  In $$\log p\_{\theta}(x^1,...,x^N)=D\_{KL}(q\_{\theta}(z|x^i)||p\_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i),$$ why does $p(x^1,...,x^N)$ and $q(z|x^i)$ have the same parameter $\theta?$
  
  Given that $p$ is just the probability of the observed data and $q$ is the approximation of the posterior, shouldn't they be different distributions and thus their parameters different?
  
  **Tags:** variational-autoencoder, latent-variable, evidence-lower-bound
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-07. Score: 2 | Accepted answer
  
  *I will try to answer your questions directly (but I guess I won't be able to), otherwise, this can become quite confusing, given the inconsistencies that can be found across different sources.*
  
  > In $logp\_{\theta}(x^1,...,x^N)=D\_{KL}(q\_{\theta}(z|x^i)||p\_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$ why is $\theta$ and param for $p$ and $q$?
  
  In a few words, your equation is wrong because it uses the letters $\phi$ and $\theta$ inconsistently.
  
  If you look more carefully at the **right-hand side** of your equation, you will notice that $q\_{\theta}$ has different parameters, i.e. $\theta$, than $p\_{\phi}$, which has parameters $\phi$, so $p$ and $q$ have different parameters, and this should be the case, because they are represented by different neural networks in the case of the VAE. **However**, the **left-hand side** uses $\theta$ as the parameters of $p$ (while the right-hand side uses $\phi$ to index $p$), so this should already suggest that **the equation is not correct** (as you correctly thought).
  
  In the case of the VAE, $\phi$ **usually** represents the parameters (or weights) of the encoder neural network (NN), while $\theta$ usually represents the parameters of the decoder NN (or vice-versa, but you should just be consistent, which is often not the case in your equation). In fact, in [the VAE paper](https://arxiv.org/pdf/1312.6114.pdf), in **equation 3**, the authors use $\phi$ to represent the parameters of the encoder $q$, while $\theta$ is used to denote the parameters of the decoder $p$.
  
  So, if you follow the notation in the VAE paper, the ELBO can actually be written something like
  
  \begin{align}
  \mathcal{L}(\phi,\theta; \mathbf{x})
  &=
  \mathbb{E}\_{\tilde{z} \sim q\_{\phi}(\mathbf{z} \mid \mathbf{x})} \left[ \log p\_{\theta} (\mathbf{x} \mid \mathbf{z}) \right] - \operatorname{KL} \left(q\_{\phi}(\mathbf{z} \mid \mathbf{x}) \| p\_{\theta}(\mathbf{z}) \right) \tag{1} \label{1}
  \end{align}
  
  The ELBO loss $\mathcal{L}(\phi,\theta; \mathbf{x})$ has both parameters (of the encoder and decoder), which will be optimized jointly. Note that I have ignored the indices in the observations $\mathbf{x}$ (for simplicity), while, in [the VAE paper](https://arxiv.org/pdf/1312.6114.pdf), they are present. Furthermore, note that, both in \ref{1} and in [the VAE paper](https://arxiv.org/pdf/1312.6114.pdf), we use bold letters (because these objects are usually vectors), i.e. $\mathbf{x}$ and $\mathbf{z}$, rather than $x$ and $z$ (like in your equation).
  
  Note also that, even though $p\_{\theta}(\mathbf{z})$ is indexed by $\theta$, in reality, this may be an un-parametrized distribution (e.g. a Gaussian with mean $0$ and variance $1$), i.e. not a family of distributions. The use of the index $\theta$ in $p\_{\theta}(\mathbf{z})$ comes from the (implicit) assumption that both $p\_{\theta}(\mathbf{z})$ and $p\_{\theta} (\mathbf{x} \mid \mathbf{z})$ come from the same family of distributions (e.g. a family of Gaussians). In fact, if you consider the family of all Gaussian distributions, then $p\_{\theta}(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I})$ also belongs to that family. But $\theta$ and $\phi$ are also used to denote the parameters (or weights) of the networks, so this becomes understanbly confusing. (To understand equation 10 of the VAE paper, see [this answer](https://ai.stackexchange.com/a/26408/2444)).
  
  > Why does $p(x^1,...,x^N)$ and $q(z|x^i)$ have the same parameter $\theta?$
  
  This is **wrong**, in fact. If you look at equation 1 of [the VAE paper](https://arxiv.org/pdf/1312.6114.pdf), they use $\theta$ to denote the parameters of $p(\mathbf{x})$, i.e. $p\_{\theta}(\mathbf{x})$, while the parameters of the encoder are $\phi$, i.e. $q\_{\phi}(\mathbf{z} \mid \mathbf{x}$).
  
  > Cause $p$ is just the probability of the observed data and $q$ is the approximation of the posterior so shouldn't they be different distributions and their parameters different?
  
  Yes.
  
  
  ---
  
  