Answers.Body: <p>At page 130 of the same book, the author states that <span class="math-container">$\hat{p}_\text{data}$</span> is an <a href="https://en.wikipedia.org/wiki/Empirical_distribution_function" rel="nofollow noreferrer">empirical distribution</a> defined by the training data. Similarly, at page 129, he states that <span class="math-container">$p_\text{data}$</span> is the true distribution that generates the set <span class="math-container">$\mathbb{X} = \{ \boldsymbol{x}^{(1)}, \dots, \boldsymbol{x}^{(m)} \}$</span>.</p>
  
  <p>What is the difference between <span class="math-container">$\hat{p}_\text{data}$</span> and <span class="math-container">$p_\text{data}$</span>? You can think of <span class="math-container">$\hat{p}_\text{data}$</span> as a histogram that is calculated from the set <span class="math-container">$\mathbb{X}$</span> and <span class="math-container">$p_\text{data}$</span> as the true distribution from which the elements in <span class="math-container">$\mathbb{X}$</span> are drawn. </p>
  
  <p>The subscript <span class="math-container">${\boldsymbol{x}, y \sim \hat{p}_\text{data}}$</span> in the expectation <span class="math-container">$\mathbb{E}_{\boldsymbol{x}, y \sim \hat{p}_\text{data}}$</span> indicates that the expectation is taken with respect to the samples drawn from the empirical distribution <span class="math-container">$\hat{p}_\text{data}$</span>. In other words, you will optimise the objective function <span class="math-container">$J$</span> using the training data. Have a look at <a href="https://stats.stackexchange.com/q/72613/82135">this question</a> for more info.</p>
  
  <p>The subscript <span class="math-container">${\boldsymbol{x}, y \sim p_\text{data}}$</span> in the expectation of formula <span class="math-container">$5.101$</span> is a typo. In fact, in this <a href="https://www.deeplearningbook.org/contents/ml.html" rel="nofollow noreferrer">online version of the book</a>, at page 151, the subscript of the expectation is  <span class="math-container">${\boldsymbol{x}, y \sim \hat{p}_\text{data}}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-25T14:14:00.083Z
Answers.Text: Thank you a lot. your answer go just straight to my confusion :D
Answers.UserDisplayName: David Ng
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-25T12:55:12.567Z
Answers.LastActivityDate: 2019-05-25T13:00:33.477Z
Answers.LastEditDate: 2019-05-25T13:00:33.477Z
Answers.UserDisplayName: nbro
Body: <p>In chapter 5 of Deep Learning book of Ian Goodfellow, some notations in the 
  loss function as below make me really confused. </p>
  
  <p>I tried to understand <span class="math-container">$x,y \sim p_{data}$</span> means a sample <span class="math-container">$(x, y)$</span> sampled from original dataset distribution (or <span class="math-container">$y$</span> is the ground truth label).
  The loss function in formula 5.101 seems to be correct for my understanding. Actually, the formula 5.101 is derived from 5.100 by adding the regularization. </p>
  
  <p>Therefore, the notation <span class="math-container">$x,y \sim \hat{p}_{data}$</span> in formula 5.96 and 5.100 is really confusing to me whether the loss function is defined correctly (kinda typo error or not). If not so, could you help me to refactor the meaning of two notations, are they similar and correct?</p>
  
  <p><a href="https://i.sstatic.net/CnAr7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CnAr7.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.sstatic.net/Zzp5P.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Zzp5P.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.sstatic.net/FIDcS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FIDcS.png" alt="enter image description here"></a>
  Many thanks for your help.</p>
  
Tags: <machine-learning><deep-learning><notation>
Title: Being confused of distribution notations in Deep Learning book
UserDisplayName: David Ng
ThreadText: # Being confused of distribution notations in Deep Learning book
  
  (Asked by: username_0 on 2019-05-25)
  
  In chapter 5 of Deep Learning book of Ian Goodfellow, some notations in the
  loss function as below make me really confused.
  
  I tried to understand $x,y \sim p\_{data}$ means a sample $(x, y)$ sampled from original dataset distribution (or $y$ is the ground truth label).
  The loss function in formula 5.101 seems to be correct for my understanding. Actually, the formula 5.101 is derived from 5.100 by adding the regularization.
  
  Therefore, the notation $x,y \sim \hat{p}\_{data}$ in formula 5.96 and 5.100 is really confusing to me whether the loss function is defined correctly (kinda typo error or not). If not so, could you help me to refactor the meaning of two notations, are they similar and correct?
  
  [![enter image description here](https://i.sstatic.net/CnAr7.png)](https://i.sstatic.net/CnAr7.png)
  
  [![enter image description here](https://i.sstatic.net/Zzp5P.png)](https://i.sstatic.net/Zzp5P.png)
  
  [![enter image description here](https://i.sstatic.net/FIDcS.png)](https://i.sstatic.net/FIDcS.png)
  Many thanks for your help.
  
  **Tags:** machine-learning, deep-learning, notation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-25. Score: 3 | Accepted answer
  
  At page 130 of the same book, the author states that $\hat{p}\_\text{data}$ is an [empirical distribution](https://en.wikipedia.org/wiki/Empirical_distribution_function) defined by the training data. Similarly, at page 129, he states that $p\_\text{data}$ is the true distribution that generates the set $\mathbb{X} = \{ \boldsymbol{x}^{(1)}, \dots, \boldsymbol{x}^{(m)} \}$.
  
  What is the difference between $\hat{p}\_\text{data}$ and $p\_\text{data}$? You can think of $\hat{p}\_\text{data}$ as a histogram that is calculated from the set $\mathbb{X}$ and $p\_\text{data}$ as the true distribution from which the elements in $\mathbb{X}$ are drawn.
  
  The subscript ${\boldsymbol{x}, y \sim \hat{p}\_\text{data}}$ in the expectation $\mathbb{E}\_{\boldsymbol{x}, y \sim \hat{p}\_\text{data}}$ indicates that the expectation is taken with respect to the samples drawn from the empirical distribution $\hat{p}\_\text{data}$. In other words, you will optimise the objective function $J$ using the training data. Have a look at [this question](https://stats.stackexchange.com/q/72613/82135) for more info.
  
  The subscript ${\boldsymbol{x}, y \sim p\_\text{data}}$ in the expectation of formula $5.101$ is a typo. In fact, in this [online version of the book](https://www.deeplearningbook.org/contents/ml.html), at page 151, the subscript of the expectation is ${\boldsymbol{x}, y \sim \hat{p}\_\text{data}}$.
  
  > **Comments:**
  > **username_0**: Thank you a lot. your answer go just straight to my confusion :D  
  
  ---
  
  