Answers.Body: <p>You're forgetting that <span class="math-container">$x$</span> can assume several values with different probability. Let's say that <span class="math-container">$x$</span> represent the roll of a fair dice. Then <span class="math-container">$p(x)$</span> will be 1/6 for all six possible values of <span class="math-container">$x$</span>.</p>
  <p><span class="math-container">$$ p(\theta|x)=\frac{p(x|\theta)p(\theta)}{p(x)} $$</span></p>
  <p>If you rearrange the formula it becomes clear that the whole point of Bayes theorem is to say that we want matching prior <span class="math-container">$p(\theta)$</span> and posterior <span class="math-container">$p(\theta|x)$</span> distributions.</p>
  <p><span class="math-container">$$ \frac{p(\theta|x)}{p(\theta)}=1=\frac{p(x|\theta)}{p(x)} $$</span></p>
  <p>cause if the left side holds, then it means that the likelihood predicted by our parameters <span class="math-container">$p(x|\theta)$</span> is equal to the real observed probability <span class="math-container">$p(x)$</span>.
  So if we train a model on some observed dice rolls, we expect a perfect model to learn for each face of the dice the real probability <span class="math-container">$p(x)$</span> which is 1/6 and not 1.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-04T17:22:31.447Z
Answers.Text: Thank you for the explanation. I guess the short answer is that contrary to the explanations I've read, here $x$ is NOT assumed to be observed, but is treated as a random variable. (btw. in your second formula, surely you didn't mean that the ratios are equal to 1 - it's only when $\theta$ and $x$ are independent).
Answers.UserDisplayName: Abrrval
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-03T21:27:47.113Z
Answers.Text: I guess it is semantics on what we mean by 'observed'. Say we rolled a dice and obtained the number 6, then we have simulated a realisation from our random variable $X$ and _observed_ the outcome of this one particular roll of the die. In the literature, when they say $x$ is observed, they typically mean that the random variable $X$ relates to something that we can observe (as opposed to the latent variable, which, by definition, is unobservable). So, whilst we have _observed_ that the outcome of this one roll of the die was 6, it does not mean it will always be the case ...
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-03T21:28:27.060Z
Answers.Text: ... and so we use the probability of the outcome of that roll when doing variational inference.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-04T15:55:18.470Z
Answers.LastActivityDate: 2022-08-04T15:55:18.470Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>Every explanation of variational inference starts with the same basic premise: given an observed variable <span class="math-container">$x$</span>, and a latent variable <span class="math-container">$z$</span>,</p>
  <p><span class="math-container">$$ p(z|x)=\frac{p(x,z)}{p(x)} $$</span></p>
  <p>and then proceeds to expand <span class="math-container">$p(x)$</span> as an expectation over <span class="math-container">$z$</span>:</p>
  <p><span class="math-container">$$ p(x) = \int{p(x,z)dz} $$</span></p>
  <p>and then states that it's too difficult to evaluate.</p>
  <p>My very very basic question is why is <span class="math-container">$p(x)$</span> not simply equal to 1? It's an observed variable!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-23T00:00:09.550Z
Comments.Text: It's a blunder to assume that just because a specific coin toss resulted in heads that necessarily implies pr(heads) = 1; in every non-trivial setting, there is some probability of tails.
Comments.UserDisplayName: Sycorax
Tags: <machine-learning><probability-theory><bayes-theorem><variational-inference>
Title: Why isn't the evidence $p(x) = 1$ if it's an observed variable?
UserDisplayName: Abrrval
ThreadText: # Why isn't the evidence $p(x) = 1$ if it's an observed variable?
  
  (Asked by: username_0 on 2022-08-04)
  
  Every explanation of variational inference starts with the same basic premise: given an observed variable $x$, and a latent variable $z$,
  
  $$ p(z|x)=\frac{p(x,z)}{p(x)} $$
  
  and then proceeds to expand $p(x)$ as an expectation over $z$:
  
  $$ p(x) = \int{p(x,z)dz} $$
  
  and then states that it's too difficult to evaluate.
  
  My very very basic question is why is $p(x)$ not simply equal to 1? It's an observed variable!
  
  **Tags:** machine-learning, probability-theory, bayes-theorem, variational-inference
  
  > **Comments:**
  > **username_1**: It's a blunder to assume that just because a specific coin toss resulted in heads that necessarily implies pr(heads) = 1; in every non-trivial setting, there is some probability of tails.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-08-04. Score: 0
  
  You're forgetting that $x$ can assume several values with different probability. Let's say that $x$ represent the roll of a fair dice. Then $p(x)$ will be 1/6 for all six possible values of $x$.
  
  $$ p(\theta|x)=\frac{p(x|\theta)p(\theta)}{p(x)} $$
  
  If you rearrange the formula it becomes clear that the whole point of Bayes theorem is to say that we want matching prior $p(\theta)$ and posterior $p(\theta|x)$ distributions.
  
  $$ \frac{p(\theta|x)}{p(\theta)}=1=\frac{p(x|\theta)}{p(x)} $$
  
  cause if the left side holds, then it means that the likelihood predicted by our parameters $p(x|\theta)$ is equal to the real observed probability $p(x)$.
  So if we train a model on some observed dice rolls, we expect a perfect model to learn for each face of the dice the real probability $p(x)$ which is 1/6 and not 1.
  
  > **Comments:**
  > **username_0**: Thank you for the explanation. I guess the short answer is that contrary to the explanations I've read, here $x$ is NOT assumed to be observed, but is treated as a random variable. (btw. in your second formula, surely you didn't mean that the ratios are equal to 1 - it's only when $\theta$ and $x$ are independent).  
  > **username_3**: I guess it is semantics on what we mean by 'observed'. Say we rolled a dice and obtained the number 6, then we have simulated a realisation from our random variable $X$ and _observed_ the outcome of this one particular roll of the die. In the literature, when they say $x$ is observed, they typically mean that the random variable $X$ relates to something that we can observe (as opposed to the latent variable, which, by definition, is unobservable). So, whilst we have _observed_ that the outcome of this one roll of the die was 6, it does not mean it will always be the case ...  
  > **username_3**: ... and so we use the probability of the outcome of that roll when doing variational inference.  
  
  ---
  
  