Answers.Body: <p>When we say that we have <span class="math-container">$N$</span> points that were <em>"drawn from the probability distribution or probability density"</em>, this means that every point <span class="math-container">$x_n$</span> had the correct probability <span class="math-container">$p(x_n)$</span> of being sampled from the distribution when we were sampling our <span class="math-container">$n^{th}$</span> point.</p>
  
  <p>For example, suppose that we wish to compute/estimate the expected value of a distribution given by flipping a coin, which is weighted such that it lands on heads two-thirds of the time (<em>value of heads</em> <span class="math-container">$= 1$</span>), and tails one-thirds of the time (<em>value of tails</em> <span class="math-container">$= 0$</span>). This means that the ground-truth probabilities are:</p>
  
  <ul>
  <li><span class="math-container">$p(1) = \frac{2}{3}$</span></li>
  <li><span class="math-container">$p(0) = \frac{1}{3}$</span></li>
  </ul>
  
  <p>In this case, we actually wouldn't have to estimate anything by sampling; the exact probabilities are known, so we could just compute the expected value as <span class="math-container">$p(1) \times 1 + p(0) \times 0 = \frac{2}{3}$</span>.</p>
  
  <p>But now suppose that we do not know exactly how the coin is weighted, i.e. we do not know the exact values of <span class="math-container">$p(1)$</span> and <span class="math-container">$p(0)$</span>. If we simply flip our weighted coin <span class="math-container">$N = 100$</span> times, we will in expectation find about <span class="math-container">$67$</span> samples of <span class="math-container">$x_n = 1$</span>, and about <span class="math-container">$33$</span> samples of <span class="math-container">$x_n = 0$</span> (rounded to integers because we can't obtain a non-integer number of observations). So we cannot explicitly use the probabilities (because we do not know them), but they will implicitly be present in the number of repeated observations we have for every possible datapoint. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-26T11:20:11.610Z
Answers.Text: So you mean sampling is used to estimate probabilities, and that's why if p(1) = 2/3, we can expect 2/3rd of samples to be 1, and hence the expectation will account for the probabilities?
Answers.UserDisplayName: sage76
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-26T11:41:25.877Z
Answers.Text: @sage76 Correct
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-26T10:56:51.547Z
Answers.LastActivityDate: 2019-07-26T10:56:51.547Z
Answers.UserDisplayName: Dennis Soemers
Body: <p><a href="https://i.sstatic.net/9bi7E.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9bi7E.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.sstatic.net/3fnEL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3fnEL.png" alt="enter image description here"></a></p>
  
  <p>This is from the book Pattern Recognition by Bishop. Why is expectation here a simple average? Why is <span class="math-container">$f(x)$</span> not being multiplied by <span class="math-container">$p(x)$</span>?</p>
  
Tags: <math><probability-distribution><expectation>
Title: Why is the expectation calculated over finite number of points drawn from a probability distribution?
UserDisplayName: sage76
ThreadText: # Why is the expectation calculated over finite number of points drawn from a probability distribution?
  
  (Asked by: username_0 on 2019-07-26)
  
  [![enter image description here](https://i.sstatic.net/9bi7E.png)](https://i.sstatic.net/9bi7E.png)
  
  [![enter image description here](https://i.sstatic.net/3fnEL.png)](https://i.sstatic.net/3fnEL.png)
  
  This is from the book Pattern Recognition by Bishop. Why is expectation here a simple average? Why is $f(x)$ not being multiplied by $p(x)$?
  
  **Tags:** math, probability-distribution, expectation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-26. Score: 1 | Accepted answer
  
  When we say that we have $N$ points that were *"drawn from the probability distribution or probability density"*, this means that every point $x\_n$ had the correct probability $p(x\_n)$ of being sampled from the distribution when we were sampling our $n^{th}$ point.
  
  For example, suppose that we wish to compute/estimate the expected value of a distribution given by flipping a coin, which is weighted such that it lands on heads two-thirds of the time (*value of heads* $= 1$), and tails one-thirds of the time (*value of tails* $= 0$). This means that the ground-truth probabilities are:
  
  * $p(1) = \frac{2}{3}$
  * $p(0) = \frac{1}{3}$
  
  In this case, we actually wouldn't have to estimate anything by sampling; the exact probabilities are known, so we could just compute the expected value as $p(1) \times 1 + p(0) \times 0 = \frac{2}{3}$.
  
  But now suppose that we do not know exactly how the coin is weighted, i.e. we do not know the exact values of $p(1)$ and $p(0)$. If we simply flip our weighted coin $N = 100$ times, we will in expectation find about $67$ samples of $x\_n = 1$, and about $33$ samples of $x\_n = 0$ (rounded to integers because we can't obtain a non-integer number of observations). So we cannot explicitly use the probabilities (because we do not know them), but they will implicitly be present in the number of repeated observations we have for every possible datapoint.
  
  > **Comments:**
  > **username_0**: So you mean sampling is used to estimate probabilities, and that's why if p(1) = 2/3, we can expect 2/3rd of samples to be 1, and hence the expectation will account for the probabilities?  
  > **username_1**: @username_0 Correct  
  
  ---
  
  