Answers.Body: <p>See the definition of <a href="https://en.wikipedia.org/wiki/Bias_(statistics)#Bias_of_an_estimator" rel="nofollow noreferrer">statistical bias</a>:</p>
  <blockquote>
  <p>Statistical bias is a feature of a statistical technique or of its results whereby the expected value of the results differs from the true underlying quantitative parameter being estimated... The bias of a statistic <span class="math-container">${\displaystyle T}$</span> is always relative to the parameter <span class="math-container">${\displaystyle \theta }$</span> it is used to estimate, but the parameter <span class="math-container">${\displaystyle \theta }$</span> is often omitted when it is clear from the context what is being estimated.</p>
  </blockquote>
  <p>Therefore the <em>average</em> in this definition is with respect to different possible training datasets that could be drawn from the same underlying data distribution. In other words, if you were to train your model multiple times on different samples from the same population, the bias is the difference between the average of these predictions and the true value.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-30T21:13:51.080Z
Answers.LastActivityDate: 2024-09-30T21:13:51.080Z
Answers.UserDisplayName: cinch
Body: <p>In many places, we have the following definition for bias:</p>
  <p><a href="https://i.sstatic.net/tr7VZXxy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tr7VZXxy.png" alt="enter image description here" /></a></p>
  <p>Average with respect to what in this definition?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-30T13:34:51.080Z
Comments.Text: depends on the model this refers to, but in linear regression, if I'm not mistaken, is set to be $\hat{y} - \beta \hat{x}$ where $\hat{y}$ is the average "target" in the dataset and $\hat{x}$ is the average value for each feature
Comments.UserDisplayName: Alberto
Tags: <machine-learning>
Title: Average with respect to what in the definition of bias in ML?
UserDisplayName: DSPinfinity
ThreadText: # Average with respect to what in the definition of bias in ML?
  
  (Asked by: username_0 on 2024-09-29)
  
  In many places, we have the following definition for bias:
  
  [![enter image description here](https://i.sstatic.net/tr7VZXxy.png)](https://i.sstatic.net/tr7VZXxy.png)
  
  Average with respect to what in this definition?
  
  **Tags:** machine-learning
  > **Comments:**
  > **username_1**: depends on the model this refers to, but in linear regression, if I'm not mistaken, is set to be $\hat{y} - \beta \hat{x}$ where $\hat{y}$ is the average "target" in the dataset and $\hat{x}$ is the average value for each feature  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-09-30. Score: 3 | Accepted answer
  
  See the definition of [statistical bias](https://en.wikipedia.org/wiki/Bias_(statistics)#Bias_of_an_estimator):
  
  > Statistical bias is a feature of a statistical technique or of its results whereby the expected value of the results differs from the true underlying quantitative parameter being estimated... The bias of a statistic ${\displaystyle T}$ is always relative to the parameter ${\displaystyle \theta }$ it is used to estimate, but the parameter ${\displaystyle \theta }$ is often omitted when it is clear from the context what is being estimated.
  
  Therefore the *average* in this definition is with respect to different possible training datasets that could be drawn from the same underlying data distribution. In other words, if you were to train your model multiple times on different samples from the same population, the bias is the difference between the average of these predictions and the true value.
  
  
  ---
  
  