Answers.Body: <p>The idea behind this kind of reasoning is that there is a &quot;true&quot; distribution (unknown to us, mere mortals) and that the data is generated following this distribution. But what we don't really know the shape of the distribution, all we know is the distribution of the data that we have. This is called the empirical distribution.   Let's see a simple example to illustrate the point.<br />
  Let's consider a die. Each number is equally likely to show up if we throw the die, so the true underlying distribution is uniform over the set <span class="math-container">$\{1, 2,\dots6\}$</span>. Now let's say you ask your friend to throw the die 60 times, what you will see is likely something close to uniform over the set <span class="math-container">$\{1, 2,\dots6\}$</span> (not really uniform though, as that would be highly unlikely). This distribution is the empirical one, and as you collect more and more samples it will converge to the actual underlying distribution.<br />
  In your case what happened is the following:<br />
  <span class="math-container">$x_1,\dots, x_m$</span> is your sample (in the example above, the <span class="math-container">$60$</span> numbers that you see as your friend throws the die). This sample defines a distribution, <span class="math-container">$\hat{p}_{data}$</span>. In the example above, <span class="math-container">$\hat{p}_{data}$</span> would likely be close to the uniform distribution over <span class="math-container">$\{1, \dots, 6\}$</span>. Now you can think about the sum over <span class="math-container">$x_1\dots, x_m$</span> of <span class="math-container">$\log(p_{model}(x^{(i)};\theta))$</span> as the average of <span class="math-container">$\log(p_{model}(x;\theta))$</span>, where <span class="math-container">$x$</span> is drawn according to <span class="math-container">$\hat{p}_{data}$</span>.<br />
  Let me make another example with some actual numbers. Let's say you toss a fair coin <span class="math-container">$7$</span> times and you see <span class="math-container">$$\{H, H, H, T, H, T, H\}.$$</span>
  The empirical distribution is <span class="math-container">$\mathbb{P}(H)=5/7$</span>, <span class="math-container">$\mathbb{P}(T)=2/7$</span>. So if you compute the expectation <span class="math-container">$$\mathbb{E}_{x \sim \hat{p}_{data}}[\log(p_{model}(x;\theta)]$$</span> you get
  <span class="math-container">$$\log(p_{model}(H;\theta)\cdot\mathbb{P}(H) +\log(p_{model}(T;\theta)\cdot\mathbb{P}(T) = \\ \frac{5}{7}\log(p_{model}(H;\theta) +\frac{2}{7}\log(p_{model}(T;\theta)),$$</span>
  which is what you will get if you compute the first sum you wrote.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-01T17:43:43.787Z
Answers.LastActivityDate: 2021-01-02T23:17:07.963Z
Answers.LastEditDate: 2021-01-02T23:17:07.963Z
Answers.UserDisplayName: Uskebasi
Body: <p>I was reading the book <a href="https://raw.githubusercontent.com/janishar/mit-deep-learning-book-pdf/master/complete-book-pdf/Ian%20Goodfellow%2C%20Yoshua%20Bengio%2C%20Aaron%20Courville%20-%20Deep%20Learning%20(2017%2C%20MIT).pdf" rel="nofollow noreferrer">Deep Learning</a> by Ian Goodfellow. I had a doubt in the Maximum likelihood estimation section (Pg 131). I understand till the Eq 5.58 which describes what is being maximized in the problem.</p>
  <p><span class="math-container">$$
   \theta_{\text{ML}} = \text{argmax}_{\theta} \sum_1^m \log(p_{\text{model}}(x^{(i)};\theta)) 
  $$</span></p>
  <p>However the next equation 5.59 restates this equation as:</p>
  <p><span class="math-container">$$
  \theta_{\text{ML}} = \text{argmax}_{\theta}  E_{x \sim \hat{p}_{\text{data}}}(\log(p_{\text{model}}(x;\theta)) 
  $$</span></p>
  <p>where <span class="math-container">$$\hat{p}_{\text{data}}$$</span> is described as the empirical distribution defined by the training data.
  Could someone explain what is meant by this empirical distribution? It seems to be different from the distribution parametrized by theta as that is described by <span class="math-container">$$ p_{\text{model}} $$</span></p>
  
Tags: <probability-distribution><maximum-likelihood>
Title: What is emperical distribution in MLE?
UserDisplayName: ANIRUDH BUVANESH
ThreadText: # What is emperical distribution in MLE?
  
  (Asked by: username_0 on 2021-01-01)
  
  I was reading the book [Deep Learning](https://raw.githubusercontent.com/janishar/mit-deep-learning-book-pdf/master/complete-book-pdf/Ian%20Goodfellow%2C%20Yoshua%20Bengio%2C%20Aaron%20Courville%20-%20Deep%20Learning%20(2017%2C%20MIT).pdf) by Ian Goodfellow. I had a doubt in the Maximum likelihood estimation section (Pg 131). I understand till the Eq 5.58 which describes what is being maximized in the problem.
  
  $$
  \theta\_{\text{ML}} = \text{argmax}\_{\theta} \sum\_1^m \log(p\_{\text{model}}(x^{(i)};\theta))
  $$
  
  However the next equation 5.59 restates this equation as:
  
  $$
  \theta\_{\text{ML}} = \text{argmax}\_{\theta} E\_{x \sim \hat{p}\_{\text{data}}}(\log(p\_{\text{model}}(x;\theta))
  $$
  
  where $$\hat{p}\_{\text{data}}$$ is described as the empirical distribution defined by the training data.
  Could someone explain what is meant by this empirical distribution? It seems to be different from the distribution parametrized by theta as that is described by $$ p\_{\text{model}} $$
  
  **Tags:** probability-distribution, maximum-likelihood
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-01. Score: 2 | Accepted answer
  
  The idea behind this kind of reasoning is that there is a "true" distribution (unknown to us, mere mortals) and that the data is generated following this distribution. But what we don't really know the shape of the distribution, all we know is the distribution of the data that we have. This is called the empirical distribution. Let's see a simple example to illustrate the point.  
  Let's consider a die. Each number is equally likely to show up if we throw the die, so the true underlying distribution is uniform over the set $\{1, 2,\dots6\}$. Now let's say you ask your friend to throw the die 60 times, what you will see is likely something close to uniform over the set $\{1, 2,\dots6\}$ (not really uniform though, as that would be highly unlikely). This distribution is the empirical one, and as you collect more and more samples it will converge to the actual underlying distribution.  
  In your case what happened is the following:  
  $x\_1,\dots, x\_m$ is your sample (in the example above, the $60$ numbers that you see as your friend throws the die). This sample defines a distribution, $\hat{p}\_{data}$. In the example above, $\hat{p}\_{data}$ would likely be close to the uniform distribution over $\{1, \dots, 6\}$. Now you can think about the sum over $x\_1\dots, x\_m$ of $\log(p\_{model}(x^{(i)};\theta))$ as the average of $\log(p\_{model}(x;\theta))$, where $x$ is drawn according to $\hat{p}\_{data}$.  
  Let me make another example with some actual numbers. Let's say you toss a fair coin $7$ times and you see $$\{H, H, H, T, H, T, H\}.$$
  The empirical distribution is $\mathbb{P}(H)=5/7$, $\mathbb{P}(T)=2/7$. So if you compute the expectation $$\mathbb{E}\_{x \sim \hat{p}\_{data}}[\log(p\_{model}(x;\theta)]$$ you get
  $$\log(p\_{model}(H;\theta)\cdot\mathbb{P}(H) +\log(p\_{model}(T;\theta)\cdot\mathbb{P}(T) = \\ \frac{5}{7}\log(p\_{model}(H;\theta) +\frac{2}{7}\log(p\_{model}(T;\theta)),$$
  which is what you will get if you compute the first sum you wrote.
  
  
  ---
  
  