Answers.Body: <p>In machine learning, the term <strong>bias</strong> can refer to at least 2 related concepts</p>
  <ol>
  <li><p>A (learnable) <strong>parameter</strong> of a model, such as a linear regression model, which allows you to learn a shifted function. For example, in the case of a linear regression model <span class="math-container">$y = f(x) = mx + b$</span>, the bias <span class="math-container">$b$</span> allows you to shift the straight-line up an down: without the bias, you would only be able to control the slope <span class="math-container">$m$</span> of the straight-line. Similarly, in a neural network, you can have a neuron that performs a linear combination of the inputs, then it uses a bias term to shift the straight-line, and you could also use the bias after having applied the activation function, but this will have a different effect.</p>
  </li>
  <li><p>Anything that <strong>guides</strong> the learning algorithm (e.g. gradient descent with back-propagation) towards a <em>specific</em> set of solutions. For example, if you use <strong>regularization</strong>, you are biasing the learning algorithm to choose, typically, smoother or simpler functions. The bias term in the linear regression model is also a way of biasing the learning algorithm: you assume that the straight-line function does not necessarily go through zero, and this assumption affects the type of functions that you can learn (and this is why these two concepts of <em>bias</em> are related!). So, there are many ways of biasing a learning algorithm.</p>
  </li>
  </ol>
  <p>The bias does not always lead you to the best solutions and can actually lead to the wrong solutions, but it is often useful in many ways, for example, it can speed up learning, as you restrict the number of functions that can be learned and searched. The bias (as described in point 2) is often discussed in the context of the <a href="https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff" rel="nofollow noreferrer">bias-variance trade-off</a>, and both the <em>bias</em> and the <em>variance</em> are related to the concepts of <em>generalization, under-fitting</em>, and <em>over-fitting</em>. The <a href="https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff" rel="nofollow noreferrer">linked Wikipedia article</a> explains these concepts quite well and provides an example of how the bias and variance are decomposed, so you should probably read that article for more details.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-26T22:22:09.300Z
Answers.LastActivityDate: 2020-09-27T13:47:07.790Z
Answers.LastEditDate: 2020-09-27T13:47:07.790Z
Answers.UserDisplayName: nbro
Body: <p>Here is a linear regression model</p>
  <p><span class="math-container">$$y = mx + b,$$</span></p>
  <p>where <span class="math-container">$b$</span> is known as <span class="math-container">$y$</span>-intercept, but also known as the <strong>bias</strong> [<a href="https://developers.google.com/machine-learning/crash-course/descending-into-ml/linear-regression" rel="noreferrer">1</a>], <span class="math-container">$m$</span> is the slope, and <span class="math-container">$x$</span> is the feature vector.</p>
  <p>As I understood, in machine learning, there is also the <strong>bias</strong> that can cause the model to underfit.</p>
  <p><em>So, is there a connection between the bias term <span class="math-container">$b$</span> in a linear regression model and the bias that can lead to under-fitting in machine learning?</em></p>
  
Tags: <machine-learning><linear-regression><bias-variance-tradeoff><underfitting><inductive-bias>
Title: Is there a connection between the bias term in a linear regression model and the bias that can lead to under-fitting?
UserDisplayName: Sivaram Rasathurai
ThreadText: # Is there a connection between the bias term in a linear regression model and the bias that can lead to under-fitting?
  
  (Asked by: username_0 on 2020-09-26)
  
  Here is a linear regression model
  
  $$y = mx + b,$$
  
  where $b$ is known as $y$-intercept, but also known as the **bias** [[1](https://developers.google.com/machine-learning/crash-course/descending-into-ml/linear-regression)], $m$ is the slope, and $x$ is the feature vector.
  
  As I understood, in machine learning, there is also the **bias** that can cause the model to underfit.
  
  *So, is there a connection between the bias term $b$ in a linear regression model and the bias that can lead to under-fitting in machine learning?*
  
  **Tags:** machine-learning, linear-regression, bias-variance-tradeoff, underfitting, inductive-bias
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-26. Score: 7 | Accepted answer
  
  In machine learning, the term **bias** can refer to at least 2 related concepts
  
  1. A (learnable) **parameter** of a model, such as a linear regression model, which allows you to learn a shifted function. For example, in the case of a linear regression model $y = f(x) = mx + b$, the bias $b$ allows you to shift the straight-line up an down: without the bias, you would only be able to control the slope $m$ of the straight-line. Similarly, in a neural network, you can have a neuron that performs a linear combination of the inputs, then it uses a bias term to shift the straight-line, and you could also use the bias after having applied the activation function, but this will have a different effect.
  2. Anything that **guides** the learning algorithm (e.g. gradient descent with back-propagation) towards a *specific* set of solutions. For example, if you use **regularization**, you are biasing the learning algorithm to choose, typically, smoother or simpler functions. The bias term in the linear regression model is also a way of biasing the learning algorithm: you assume that the straight-line function does not necessarily go through zero, and this assumption affects the type of functions that you can learn (and this is why these two concepts of *bias* are related!). So, there are many ways of biasing a learning algorithm.
  
  The bias does not always lead you to the best solutions and can actually lead to the wrong solutions, but it is often useful in many ways, for example, it can speed up learning, as you restrict the number of functions that can be learned and searched. The bias (as described in point 2) is often discussed in the context of the [bias-variance trade-off](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff), and both the *bias* and the *variance* are related to the concepts of *generalization, under-fitting*, and *over-fitting*. The [linked Wikipedia article](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) explains these concepts quite well and provides an example of how the bias and variance are decomposed, so you should probably read that article for more details.
  
  
  ---
  
  