Answers.Body: <p>If the full joint distribution <span class="math-container">$p(o,x)=p(o|x)p(x)$</span> (usually assumed to be the product of two Gaussians for free energy principle of predictive coding, and in many practical situations you can only learn the full joint distribution alongside the variational posterior on the fly via EM algo) is allowed to calculate <span class="math-container">$p(o)$</span> directly, then you'd first have to know the posterior function <span class="math-container">$p(x|o)$</span> which is exactly what the variational inference here aims to approximate using a variational posterior <span class="math-container">$q(x|o; φ)$</span> with parameters <span class="math-container">$φ$</span> instead of the convergence slower or inefficient MCMC method, and you've already mentioned <span class="math-container">$p(o)$</span> is intractable directly as it requires an integration over all latent variable states.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-27T22:16:26.033Z
Answers.LastActivityDate: 2023-04-27T22:16:26.033Z
Answers.UserDisplayName: cinch
Body: <p>I'm struggling to understand one specific part of the formalism of the free energy principle.</p>
  <p>My understanding is that the free energy principle can be derived from considering statistical dynamics of a system that is coupled with its environment in some non-trivial way. To use a more specific instance, we can think of sensory perception as estimating a latent state given some sensory observation. Thus, the problem would be straightforwardly interpreted as Bayesian inference as the estimation of the quantity:</p>
  <p><span class="math-container">$$P(x \mid o) = \frac{P(o \mid x)p(x)}{P(o)}$$</span></p>
  <p>The denominator <span class="math-container">$P(o)$</span> is intractable as it involves integrating <span class="math-container">$P(o|x) dx$</span> over all latent variable states <span class="math-container">$x$</span>.</p>
  <p>The problem is then solved as an optimization problem of fitting the variational posterior <span class="math-container">$Q(x \mid o)$</span> to the true posterior <span class="math-container">$P(x \mid o)$</span> by minimizing the KL divergence between the distributions.</p>
  <p>The free energy then is simply the upper bound on this divergence which is the KL divergence <span class="math-container">$D(Q(x \mid o) || P(o, x))$</span>. So our upper bound -- the free energy -- is the KL divergence between variational posterior and the joint distribution.</p>
  <p>What confuses me is that it seems like knowing the joint distribution allows one to calculate the value <span class="math-container">$P(o)$</span> directly avoiding this whole minimization using the KL divergence, since <span class="math-container">$P(o, x) = P(o \mid x)P(x)$</span>.</p>
  <p>For reference: <a href="https://arxiv.org/abs/2107.12979" rel="nofollow noreferrer">https://arxiv.org/abs/2107.12979</a> (pages 7 and 8 contain the derivation I'm referring to)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-01T11:34:37.597Z
Comments.Text: Please, next time, put your specific question in the title. Thanks.
Comments.UserDisplayName: nbro
Tags: <machine-learning><evidence-lower-bound><variational-inference>
Title: If we know the joint distribution, can we simply derive the evidence from it?
UserDisplayName: Gustavo
ThreadText: # If we know the joint distribution, can we simply derive the evidence from it?
  
  (Asked by: username_0 on 2023-04-27)
  
  I'm struggling to understand one specific part of the formalism of the free energy principle.
  
  My understanding is that the free energy principle can be derived from considering statistical dynamics of a system that is coupled with its environment in some non-trivial way. To use a more specific instance, we can think of sensory perception as estimating a latent state given some sensory observation. Thus, the problem would be straightforwardly interpreted as Bayesian inference as the estimation of the quantity:
  
  $$P(x \mid o) = \frac{P(o \mid x)p(x)}{P(o)}$$
  
  The denominator $P(o)$ is intractable as it involves integrating $P(o|x) dx$ over all latent variable states $x$.
  
  The problem is then solved as an optimization problem of fitting the variational posterior $Q(x \mid o)$ to the true posterior $P(x \mid o)$ by minimizing the KL divergence between the distributions.
  
  The free energy then is simply the upper bound on this divergence which is the KL divergence $D(Q(x \mid o) || P(o, x))$. So our upper bound -- the free energy -- is the KL divergence between variational posterior and the joint distribution.
  
  What confuses me is that it seems like knowing the joint distribution allows one to calculate the value $P(o)$ directly avoiding this whole minimization using the KL divergence, since $P(o, x) = P(o \mid x)P(x)$.
  
  For reference: <https://arxiv.org/abs/2107.12979> (pages 7 and 8 contain the derivation I'm referring to)
  
  **Tags:** machine-learning, evidence-lower-bound, variational-inference
  > **Comments:**
  > **username_1**: Please, next time, put your specific question in the title. Thanks.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-04-27. Score: 1
  
  If the full joint distribution $p(o,x)=p(o|x)p(x)$ (usually assumed to be the product of two Gaussians for free energy principle of predictive coding, and in many practical situations you can only learn the full joint distribution alongside the variational posterior on the fly via EM algo) is allowed to calculate $p(o)$ directly, then you'd first have to know the posterior function $p(x|o)$ which is exactly what the variational inference here aims to approximate using a variational posterior $q(x|o; φ)$ with parameters $φ$ instead of the convergence slower or inefficient MCMC method, and you've already mentioned $p(o)$ is intractable directly as it requires an integration over all latent variable states.
  
  
  ---
  
  