Answers.Body: <p>The <strong>inductive bias</strong> is the <strong>prior knowledge</strong> that you incorporate in the learning process that biases the learning algorithm to choose from a <em>specific</em> set of functions [<a href="https://www.cs.huji.ac.il/%7Eshais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf" rel="nofollow noreferrer">1</a>].</p>
  <p>For example, if you choose the hypothesis class</p>
  <p><span class="math-container">$$\mathcal{H}_\text{lines} = \{f(x) = ax + b \mid a, b \in \mathbb{R} \}$$</span> rather than <span class="math-container">$$\mathcal{H}_\text{parabolas} = \{f(x) = ax^2 + b \mid a, b \in \mathbb{R} \},$$</span> then you're assuming (implicitly or explicitly, depending on whether you're aware of these concepts) that your <em>target function</em> (the function that you want to learn) lies in the set <span class="math-container">$\mathcal{H}_\text{lines}$</span>. If that's the case, then your learning algorithm is more likely to find it.</p>
  <p>In most cases, you do not know exactly the nature of your target function, so you could think that it may be a good idea to choose the largest set of possible functions, but this would make learning infeasible (i.e. you have too many functions to choose from) and could lead to <strong>over-fitting</strong>, i.e. you choose a function that performs well on your training data, but it's actually quite different from your target function, so it performs badly on unseen data (from your target function). This can happen because the training data could not be representative of your target function (you don't usually know this a priori, so you cannot really or completely solve this issue).</p>
  <p>So, the definition above does <em>not</em> imply that the inductive bias will not necessarily lead to over-fitting or, equivalently, will not negatively affect the generalization of your chosen function. Of course, if you chose to use a CNN (rather than an MLP) because you are dealing with images, then you will probably get better performance. However, if you mistakingly assume that your target function is linear and you choose <span class="math-container">$\mathcal{H}_\text{lines}$</span> from which your learning algorithm can pick functions, then it will choose a bad function.</p>
  <p><a href="https://www.cs.huji.ac.il/%7Eshais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf#page=36" rel="nofollow noreferrer">Section 2.3</a> of the book <a href="https://www.cs.huji.ac.il/%7Eshais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf" rel="nofollow noreferrer">Understanding Machine Learning: From Theory to Algorithms</a> and <a href="http://noiselab.ucsd.edu/ECE228/Murphy_Machine_Learning.pdf#page=50" rel="nofollow noreferrer">section 1.4.4.</a> of the book <a href="http://noiselab.ucsd.edu/ECE228/Murphy_Machine_Learning.pdf#page=50" rel="nofollow noreferrer">Machine Learning A Probabilistic Perspective</a> (by Murphy) provide more details about the inductive bias (the first more from a learning theory perspective, while the second more from a probabilistic one).</p>
  <p>You may also be interested in <a href="https://ai.stackexchange.com/a/23887/2444">this answer</a> that I wrote a while ago about the difference between approximation and estimation errors (although if you know nothing about learning theory it may not be very understandable). In any case, the idea is that the approximation error (AE) can be a synonym for inductive bias because the AE is the error due to the choice of hypothesis class.</p>
  <p>(As a side note, I think it is called &quot;inductive bias&quot; because this bias is the one that can make <em>inductive inference</em> feasible and successful [<a href="https://www.cin.ufpe.br/%7Ecavmj/Machine%20-%20Learning%20-%20Tom%20Mitchell.pdf#page=52" rel="nofollow noreferrer">2</a>] and maybe to differentiate it from other biases, e.g. the bias term in linear regression, although <a href="https://ai.stackexchange.com/a/23784/2444">that term can also be an inductive bias</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-04T18:56:21.273Z
Answers.Text: Nice answer! With regards to the "feasible and successful", I am thinking it as following: It is still possible that with $\mathcal{H}_\text{parabolas}$ we get back a linear function. Assuming that the true relation is linear what we gain by selecting $\mathcal{H}_\text{lines}$. **We can find the right relationship with less training data**. Is this correct?
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-08T03:26:36.167Z
Answers.LastActivityDate: 2021-09-08T03:26:36.167Z
Answers.UserDisplayName: nbro
Answers.Body: <blockquote>
  <p>Is it true that a bias is said to be inductive iff it is useful in generalising the data?</p>
  </blockquote>
  <blockquote>
  <p>Or does inductive bias can also refer to the assumptions that may cause a decrease in performance?</p>
  </blockquote>
  <p><a href="http://www.cs.cmu.edu/%7Etom/pubs/NeedForBias_1980.pdf" rel="nofollow noreferrer">Tom M. Mitchell</a> defines bias as:</p>
  <blockquote>
  <p>Any basis for choosing one generalization over another, other than strict consistency with the observed training instances.</p>
  </blockquote>
  <p><a href="https://link.springer.com/content/pdf/10.1023/A:1022630017346.pdf#page=2" rel="nofollow noreferrer">Gordon, Desjardins</a> extend the definition of bias to include any factor (including consistency with the instances) that influences the definition or selection of inductive hypotheses.</p>
  <p>Basically inductive bias is <strong>any type of bias</strong> that a learning algorithm introduces in order to provide a prediction.</p>
  <p>For example:</p>
  <ol>
  <li>In SVM we attempt to maximize the width of the boundary between two classes</li>
  <li>In Nearest neighbors we assume that most of the cases in a small neighborhood in feature space belong to the same class</li>
  <li>In cross-validation choosing the model that minimizes the cross-validation error is an inductive bias, because we are choosing an hypothesis over another</li>
  </ol>
  <p>However it must be said that cross-validation score has a well grounded <a href="https://web.stanford.edu/%7Ehastie/ElemStatLearn/printings/ESLII_print12_toc.pdf#page=260" rel="nofollow noreferrer">theoretical background</a>, and trying to avoid its bias by only considering a hypothesis class is itself a bias because we are discarding all the other hypothesis a priori.</p>
  <p>In your case:</p>
  <blockquote>
  <p>Suppose I have a dataset on which I want to use a deep neural network to do my task. I think, based on some knowledge, that a DNN with 5 or 11 layers may work well. But, after implementation, suppose 11 layer one worked well. Then can I call both of them inductive bias? or only the 11 layer assumption?</p>
  </blockquote>
  <p>In this case you have introduced an inductive bias by using a Neural network between 5 and 11 layers (you chose a function class) then you introduced another bias by choosing the number of layers that minimizes the cross-validation score.</p>
  <h1>Conclusion</h1>
  <p>In the end we want to minimize the error on the real-world task that we need to solve, for example maximizing the accuracy on MNIST. To do this we use certain hypothesis/models, and methods to compare them, i.e. cross-validation score. The choice of the models and methods are up to the researcher and its prior beliefs. All these beliefs result in choices about the learning algorithm and introduce an inductive bias.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-07T16:59:13.600Z
Answers.Text: You're quoting a Wikipedia article that does not cite any external resources (e.g. for the part of cross-validation that you quote). Although what you wrote doesn't seem wrong to me, I would encourage you to search for more reliable resources (e.g. reliable books).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-08T13:06:51.567Z
Answers.Text: @nbro Thank you for pointing that out, I edited the answer by including better references and, I think, a better definition of inductive bias
Answers.UserDisplayName: LetteraUnica
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-07T16:29:56.303Z
Answers.LastActivityDate: 2021-09-08T13:04:12.257Z
Answers.LastEditDate: 2021-09-08T13:04:12.257Z
Answers.UserDisplayName: LetteraUnica
Answers.Body: <p>The inductive bias assumed by CNN is that if we translate an image, the output does not change (the image has translational symmetry), and we can see that this assumption is valid. Similarly, spherical CNN  has rotational symmetry as inductive bias capture by the SO3 group (a collection of all the special orthogonal <span class="math-container">$3 \times 3$</span> matrices), and this is valid when data is on a sphere. The inductive bias of linear regression is that the data can be separated linearly. </p>
  <p>We must choose algorithms such that the inductive bias captures the correct assumption about the data distribution. For example, linear regression is better than polynomial regression if data is perfectly linear, since the inductive bias of linear regression matches with the data distribution. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-07T20:29:19.330Z
Answers.Text: So, can you conclude that the inductive bias doesn't necessarily lead to better "performance"?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-08T07:37:26.440Z
Answers.Text: Inductive bias leads to better generalization for example, we use CNN for image classification rather than NN, NN will need lot of translational images to identify that image has translational symmetry, but CNN already assume translation symmetry(inductive bias of CNN). It will also cause curse of dimensionality in case of NN and we may not have enough sample to learn that symmetry.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-08T07:41:06.870Z
Answers.Text: If the inductive bias goes with the assumption of the data, then it leads to better reasult. For example, CNN is better than linear regression for images since CNN inductive bias captures the underlying assumption but linear regression's inductive bias does not go with the assumption
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-07T21:19:05.087Z
Answers.Text: @SwaksharDeb Does inductive bias afftect the bias or variance term in the bias-variance decomposition?
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-07T19:08:38.353Z
Answers.LastActivityDate: 2021-09-08T02:31:45.533Z
Answers.LastEditDate: 2021-09-08T02:31:45.533Z
Answers.UserDisplayName: Swakshar Deb
Body: <p>Is it true that <strong>a bias is said to be inductive iff it is <em>useful</em> in generalising the data</strong>?</p>
  <p>Or does inductive bias can also refer to the assumptions that may cause a decrease in performance?</p>
  <hr />
  <p>Suppose I have a dataset on which I want to use a deep neural network to do my task. I think, based on some knowledge, that a DNN with 5 or 11 layers may work well. But, after implementation, suppose 11 layer one worked well. Then can I call both of them inductive bias? or only the 11 layer assumption?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-08T01:31:10.567Z
Comments.Text: @nbro It seems that any type of bias can be treated as an inductive bias from both the answers.
Comments.UserDisplayName: hanugm
Tags: <machine-learning><terminology><generalization><inductive-bias>
Title: Is the inductive bias always a useful bias for generalisation?
UserDisplayName: hanugm
ThreadText: # Is the inductive bias always a useful bias for generalisation?
  
  (Asked by: username_0 on 2021-09-07)
  
  Is it true that **a bias is said to be inductive iff it is *useful* in generalising the data**?
  
  Or does inductive bias can also refer to the assumptions that may cause a decrease in performance?
  
  ---
  
  Suppose I have a dataset on which I want to use a deep neural network to do my task. I think, based on some knowledge, that a DNN with 5 or 11 layers may work well. But, after implementation, suppose 11 layer one worked well. Then can I call both of them inductive bias? or only the 11 layer assumption?
  
  **Tags:** machine-learning, terminology, generalization, inductive-bias
  > **Comments:**
  > **username_0**: @nbro It seems that any type of bias can be treated as an inductive bias from both the answers.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2021-09-08. Score: 3
  
  The **inductive bias** is the **prior knowledge** that you incorporate in the learning process that biases the learning algorithm to choose from a *specific* set of functions [[1](https://www.cs.huji.ac.il/%7Eshais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf)].
  
  For example, if you choose the hypothesis class
  
  $$\mathcal{H}\_\text{lines} = \{f(x) = ax + b \mid a, b \in \mathbb{R} \}$$ rather than $$\mathcal{H}\_\text{parabolas} = \{f(x) = ax^2 + b \mid a, b \in \mathbb{R} \},$$ then you're assuming (implicitly or explicitly, depending on whether you're aware of these concepts) that your *target function* (the function that you want to learn) lies in the set $\mathcal{H}\_\text{lines}$. If that's the case, then your learning algorithm is more likely to find it.
  
  In most cases, you do not know exactly the nature of your target function, so you could think that it may be a good idea to choose the largest set of possible functions, but this would make learning infeasible (i.e. you have too many functions to choose from) and could lead to **over-fitting**, i.e. you choose a function that performs well on your training data, but it's actually quite different from your target function, so it performs badly on unseen data (from your target function). This can happen because the training data could not be representative of your target function (you don't usually know this a priori, so you cannot really or completely solve this issue).
  
  So, the definition above does *not* imply that the inductive bias will not necessarily lead to over-fitting or, equivalently, will not negatively affect the generalization of your chosen function. Of course, if you chose to use a CNN (rather than an MLP) because you are dealing with images, then you will probably get better performance. However, if you mistakingly assume that your target function is linear and you choose $\mathcal{H}\_\text{lines}$ from which your learning algorithm can pick functions, then it will choose a bad function.
  
  [Section 2.3](https://www.cs.huji.ac.il/%7Eshais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf#page=36) of the book [Understanding Machine Learning: From Theory to Algorithms](https://www.cs.huji.ac.il/%7Eshais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf) and [section 1.4.4.](http://noiselab.ucsd.edu/ECE228/Murphy_Machine_Learning.pdf#page=50) of the book [Machine Learning A Probabilistic Perspective](http://noiselab.ucsd.edu/ECE228/Murphy_Machine_Learning.pdf#page=50) (by Murphy) provide more details about the inductive bias (the first more from a learning theory perspective, while the second more from a probabilistic one).
  
  You may also be interested in [this answer](https://ai.stackexchange.com/a/23887/2444) that I wrote a while ago about the difference between approximation and estimation errors (although if you know nothing about learning theory it may not be very understandable). In any case, the idea is that the approximation error (AE) can be a synonym for inductive bias because the AE is the error due to the choice of hypothesis class.
  
  (As a side note, I think it is called "inductive bias" because this bias is the one that can make *inductive inference* feasible and successful [[2](https://www.cin.ufpe.br/%7Ecavmj/Machine%20-%20Learning%20-%20Tom%20Mitchell.pdf#page=52)] and maybe to differentiate it from other biases, e.g. the bias term in linear regression, although [that term can also be an inductive bias](https://ai.stackexchange.com/a/23784/2444)).
  
  > **Comments:**
  > **username_2**: Nice answer! With regards to the "feasible and successful", I am thinking it as following: It is still possible that with $\mathcal{H}_\text{parabolas}$ we get back a linear function. Assuming that the true relation is linear what we gain by selecting $\mathcal{H}_\text{lines}$. **We can find the right relationship with less training data**. Is this correct?  
  
  ---
  
  ### Answer by username_3 on 2021-09-07. Score: 1
  
  > Is it true that a bias is said to be inductive iff it is useful in generalising the data?
  
  > Or does inductive bias can also refer to the assumptions that may cause a decrease in performance?
  
  [Tom M. Mitchell](http://www.cs.cmu.edu/%7Etom/pubs/NeedForBias_1980.pdf) defines bias as:
  
  > Any basis for choosing one generalization over another, other than strict consistency with the observed training instances.
  
  [Gordon, Desjardins](https://link.springer.com/content/pdf/10.1023/A:1022630017346.pdf#page=2) extend the definition of bias to include any factor (including consistency with the instances) that influences the definition or selection of inductive hypotheses.
  
  Basically inductive bias is **any type of bias** that a learning algorithm introduces in order to provide a prediction.
  
  For example:
  
  1. In SVM we attempt to maximize the width of the boundary between two classes
  2. In Nearest neighbors we assume that most of the cases in a small neighborhood in feature space belong to the same class
  3. In cross-validation choosing the model that minimizes the cross-validation error is an inductive bias, because we are choosing an hypothesis over another
  
  However it must be said that cross-validation score has a well grounded [theoretical background](https://web.stanford.edu/%7Ehastie/ElemStatLearn/printings/ESLII_print12_toc.pdf#page=260), and trying to avoid its bias by only considering a hypothesis class is itself a bias because we are discarding all the other hypothesis a priori.
  
  In your case:
  
  > Suppose I have a dataset on which I want to use a deep neural network to do my task. I think, based on some knowledge, that a DNN with 5 or 11 layers may work well. But, after implementation, suppose 11 layer one worked well. Then can I call both of them inductive bias? or only the 11 layer assumption?
  
  In this case you have introduced an inductive bias by using a Neural network between 5 and 11 layers (you chose a function class) then you introduced another bias by choosing the number of layers that minimizes the cross-validation score.
  
  # Conclusion
  
  In the end we want to minimize the error on the real-world task that we need to solve, for example maximizing the accuracy on MNIST. To do this we use certain hypothesis/models, and methods to compare them, i.e. cross-validation score. The choice of the models and methods are up to the researcher and its prior beliefs. All these beliefs result in choices about the learning algorithm and introduce an inductive bias.
  
  > **Comments:**
  > **username_1**: You're quoting a Wikipedia article that does not cite any external resources (e.g. for the part of cross-validation that you quote). Although what you wrote doesn't seem wrong to me, I would encourage you to search for more reliable resources (e.g. reliable books).  
  > **username_3**: @nbro Thank you for pointing that out, I edited the answer by including better references and, I think, a better definition of inductive bias  
  
  ---
  
  ### Answer by username_4 on 2021-09-07. Score: 0
  
  The inductive bias assumed by CNN is that if we translate an image, the output does not change (the image has translational symmetry), and we can see that this assumption is valid. Similarly, spherical CNN  has rotational symmetry as inductive bias capture by the SO3 group (a collection of all the special orthogonal $3 \times 3$ matrices), and this is valid when data is on a sphere. The inductive bias of linear regression is that the data can be separated linearly.
  
  We must choose algorithms such that the inductive bias captures the correct assumption about the data distribution. For example, linear regression is better than polynomial regression if data is perfectly linear, since the inductive bias of linear regression matches with the data distribution.
  
  > **Comments:**
  > **username_1**: So, can you conclude that the inductive bias doesn't necessarily lead to better "performance"?  
  > **username_4**: Inductive bias leads to better generalization for example, we use CNN for image classification rather than NN, NN will need lot of translational images to identify that image has translational symmetry, but CNN already assume translation symmetry(inductive bias of CNN). It will also cause curse of dimensionality in case of NN and we may not have enough sample to learn that symmetry.  
  > **username_4**: If the inductive bias goes with the assumption of the data, then it leads to better reasult. For example, CNN is better than linear regression for images since CNN inductive bias captures the underlying assumption but linear regression's inductive bias does not go with the assumption  
  > **username_2**: @SwaksharDeb Does inductive bias afftect the bias or variance term in the bias-variance decomposition?  
  
  ---
  
  