Answers.Body: <p>As for your question why <span class="math-container">$p(z|n)\triangleq p(z)$</span>, it's the same reason for the latent prior <span class="math-container">${\displaystyle p_{\theta }(z)}$</span> of traditional <a href="https://en.wikipedia.org/wiki/Variational_autoencoder" rel="nofollow noreferrer">VAE</a> which belongs to the decoder as a regularizer and is independent of any input data <span class="math-container">$x_n$</span> and is often assumed to be the standard multivariate Gaussian for continuously valued input data such as images mainly due to analytic simplicity, sampling convenience, and noninformative continuous latent space prior. Note the encoder's latent posterior <span class="math-container">$q(z|n)$</span> is different and is dependent of the input <span class="math-container">$x_n$</span> as you quoted.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-08T06:55:09.697Z
Answers.LastActivityDate: 2024-09-08T06:55:09.697Z
Answers.UserDisplayName: cinch
Body: <p>recently I'm trying to read a paper by <a href="https://www.cs.columbia.edu/%7Eblei/fogm/2020F/readings/HoffmanJohnson2016.pdf" rel="nofollow noreferrer">Hoffman and Johnson</a> discussing an alternative decomposition of ELBO in variational autoencoders. In formula (9) and (10) of their original paper, they proposed some pre-requisite assumptions as follows:</p>
  <p><span class="math-container">\begin{equation}
  q(n, z) \triangleq q(n)q(z|n);\ q(z|n) \triangleq q(z|x_n);\ q(n)=1/N
  \end{equation}</span></p>
  <p><span class="math-container">\begin{equation}
  p(n, z) \triangleq p(n)p(z|n);\ p(z|n) \triangleq p(z);\ p(n)=1/N
  \end{equation}</span></p>
  <p>where n is the index of a randomly picked instance from a size <span class="math-container">$N$</span> subsample from the data distribution <span class="math-container">$p(x)$</span>. With these preliminaries, H&amp;J derived equation (11):</p>
  <p><span class="math-container">\begin{equation}
  \frac{1}{N}\sum_{n=1}^{N} \mathrm{KL}(q(z_n|x_n)\|p(z_n)) = \mathrm{KL}(q(z)|p(z)) + \log{N} - \mathbb{E}_{q(z)}[\mathbb{H}[q(n|z)]] 
  \end{equation}</span></p>
  <p>I had no problem deriving (11) from (9) and (10), but why <span class="math-container">$p(z|n)\triangleq p(z)$</span>? Pardon me if I have missed any critical statement in the context, but these authors seemed not to have given any justification on this formula before using it. How could one assume that the posterior probability <span class="math-container">$p(z|n)$</span> to get a specific <span class="math-container">$z$</span> given an <span class="math-container">$x_n$</span> is independent of <span class="math-container">$x_n$</span>?</p>
  <p>I will really appreciate it if you can provide some better insight! Thank you!</p>
  
Tags: <variational-autoencoder><evidence-lower-bound>
Title: Question regarding the ELBO decomposition proposed by Hoffman&Johnson
UserDisplayName: Izzy Tse
ThreadText: # Question regarding the ELBO decomposition proposed by Hoffman&Johnson
  
  (Asked by: username_0 on 2024-08-30)
  
  recently I'm trying to read a paper by [Hoffman and Johnson](https://www.cs.columbia.edu/%7Eblei/fogm/2020F/readings/HoffmanJohnson2016.pdf) discussing an alternative decomposition of ELBO in variational autoencoders. In formula (9) and (10) of their original paper, they proposed some pre-requisite assumptions as follows:
  
  \begin{equation}
  q(n, z) \triangleq q(n)q(z|n);\ q(z|n) \triangleq q(z|x\_n);\ q(n)=1/N
  \end{equation}
  
  \begin{equation}
  p(n, z) \triangleq p(n)p(z|n);\ p(z|n) \triangleq p(z);\ p(n)=1/N
  \end{equation}
  
  where n is the index of a randomly picked instance from a size $N$ subsample from the data distribution $p(x)$. With these preliminaries, H&J derived equation (11):
  
  \begin{equation}
  \frac{1}{N}\sum\_{n=1}^{N} \mathrm{KL}(q(z\_n|x\_n)\|p(z\_n)) = \mathrm{KL}(q(z)|p(z)) + \log{N} - \mathbb{E}\_{q(z)}[\mathbb{H}[q(n|z)]]
  \end{equation}
  
  I had no problem deriving (11) from (9) and (10), but why $p(z|n)\triangleq p(z)$? Pardon me if I have missed any critical statement in the context, but these authors seemed not to have given any justification on this formula before using it. How could one assume that the posterior probability $p(z|n)$ to get a specific $z$ given an $x\_n$ is independent of $x\_n$?
  
  I will really appreciate it if you can provide some better insight! Thank you!
  
  **Tags:** variational-autoencoder, evidence-lower-bound
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-08. Score: 1 | Accepted answer
  
  As for your question why $p(z|n)\triangleq p(z)$, it's the same reason for the latent prior ${\displaystyle p\_{\theta }(z)}$ of traditional [VAE](https://en.wikipedia.org/wiki/Variational_autoencoder) which belongs to the decoder as a regularizer and is independent of any input data $x\_n$ and is often assumed to be the standard multivariate Gaussian for continuously valued input data such as images mainly due to analytic simplicity, sampling convenience, and noninformative continuous latent space prior. Note the encoder's latent posterior $q(z|n)$ is different and is dependent of the input $x\_n$ as you quoted.
  
  
  ---
  
  