Answers.Body: <h1>Please note:</h1>
  <ul>
  <li>I am only referring the decision boundary to be a line for simplicity, more often than not it is a hyperplane which is difficult to visualize and spans over <code>n</code> dimensions where n is the dimensionality of your feature space.</li>
  <li>The explanation is toned in a more general way for emphasizing explainability.</li>
  </ul>
  <h1>Answer</h1>
  <blockquote>
  <p>What are the 'noisy factors' here? Does it refers to the features that are irrelevant to the class label?</p>
  </blockquote>
  <ul>
  <li>Not Necessarily.</li>
  </ul>
  <blockquote>
  <p>Or does it mean the noise/errors in the values taken by features that accidentally correlate with the class label?</p>
  </blockquote>
  <ul>
  <li>I'm not quite sure I understand.</li>
  </ul>
  <h1>However</h1>
  <p><code>Noisy factors</code> as the literature puts it are outliers in a finite data class. Imagine a dataset where we are asked to calculate the average of a set of numbers. Lets say the numbers are the set <em>S = {2,2,2,2,2,2,2,1000}</em>. The mean value in the case mentioned is 2 if it wasn't for the <code>1000</code> at the end.</p>
  <ul>
  <li>1000 could be an outlier when you are trying to approximate the mean of the set with some algorithm. The algorithm is more likely to encounter a 2 in the unseen test set than the 1000 it encountered once in a training set.</li>
  <li>When an algorithm like Linear Regressions &quot;learns&quot; something it is actually learning the weights and intercepts which modify the position of the line in the decision space.</li>
  <li>The modifications are Translation(Additive operations) and Rotation(Multiplicative operations) to the said line. It is commonly referred to as the &quot;weights&quot; and &quot;biases&quot; respectively in case of a simple line Y = mx + c where m is the <code>slope</code> and c is the <code>intercept</code>.</li>
  <li>The idea behind noise in the above text is that this:
  When an algorithm is trying to &quot;learn&quot; these weights you would want it to ignore the outliers - &quot;Generalize&quot; but you also want it to not ignore the feature data completely and introduce randomness - &quot;Specialization&quot;.</li>
  <li>How well the line is placed in your feature space is what determines your <code>Classification Effectiveness</code>.</li>
  <li>In an ideal world you would want your decision boundary to ignore such outliers which are called <code>noise</code> in the above literature.</li>
  </ul>
  <h1>Pictures (because everyone likes them)</h1>
  <ul>
  <li>In the picture below the left image is what a good decision boundary is and the right image is what an overfitted Decision boundary is.</li>
  <li>In the left-image case you are trading misclassifications at the cost of better generalizability(Consider that more likely you are going to see an x in the 2nd quadrant and the o was maybe an outlier). You do not fit your training set completely with a 100% accuracy.
  <a href="https://i.sstatic.net/LIBhh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LIBhh.png" alt="Normal Decision boundary vs Overfitted Decision Boundary" /></a></li>
  </ul>
  <h1>TL;DR</h1>
  <p>You want the model to learn that the x(s) are on the right and the o(s) are on the left but not so specifically as to which x was where. The x in the 4th quadrant and the o in the second quadrant are <code>noisy factors</code>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-17T14:12:13.037Z
Answers.LastActivityDate: 2021-06-17T14:19:56.620Z
Answers.LastEditDate: 2021-06-17T14:19:56.620Z
Answers.UserDisplayName: AvidJoe
Body: <p>Consider the following excerpt from section <strong>5.5 Regularization</strong> (p. 13) of this chapter <a href="https://web.stanford.edu/%7Ejurafsky/slp3/5.pdf" rel="nofollow noreferrer">Logistic Regression</a>.</p>
  <blockquote>
  <p>There is a problem with learning weights that make the model perfectly match the training data.  If a feature is perfectly predictive of the outcome because it happens to only occur in one class, it will be assigned a very high weight.  The weights for features will attempt to perfectly fit details of the training set, in fact too perfectly, <em>modeling <em><strong>noisy factors</strong></em> that just accidentally correlate with the class</em>. This problem is called overfitting.</p>
  </blockquote>
  <p>What are the 'noisy factors' here? Does it refer to the features that are irrelevant to the class label?</p>
  <p>Or does it mean the noise/errors in the values taken by features that accidentally correlate with the class label?</p>
  
Tags: <machine-learning><terminology><overfitting><books>
Title: What are the 'noisy factors' leading to overfitting?
UserDisplayName: hanugm
ThreadText: # What are the 'noisy factors' leading to overfitting?
  
  (Asked by: username_0 on 2021-06-17)
  
  Consider the following excerpt from section **5.5 Regularization** (p. 13) of this chapter [Logistic Regression](https://web.stanford.edu/%7Ejurafsky/slp3/5.pdf).
  
  > There is a problem with learning weights that make the model perfectly match the training data. If a feature is perfectly predictive of the outcome because it happens to only occur in one class, it will be assigned a very high weight. The weights for features will attempt to perfectly fit details of the training set, in fact too perfectly, *modeling ***noisy factors*** that just accidentally correlate with the class*. This problem is called overfitting.
  
  What are the 'noisy factors' here? Does it refer to the features that are irrelevant to the class label?
  
  Or does it mean the noise/errors in the values taken by features that accidentally correlate with the class label?
  
  **Tags:** machine-learning, terminology, overfitting, books
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-17. Score: 1
  
  # Please note:
  
  * I am only referring the decision boundary to be a line for simplicity, more often than not it is a hyperplane which is difficult to visualize and spans over `n` dimensions where n is the dimensionality of your feature space.
  * The explanation is toned in a more general way for emphasizing explainability.
  
  # Answer
  
  > What are the 'noisy factors' here? Does it refers to the features that are irrelevant to the class label?
  
  * Not Necessarily.
  
  > Or does it mean the noise/errors in the values taken by features that accidentally correlate with the class label?
  
  * I'm not quite sure I understand.
  
  # However
  
  `Noisy factors` as the literature puts it are outliers in a finite data class. Imagine a dataset where we are asked to calculate the average of a set of numbers. Lets say the numbers are the set *S = {2,2,2,2,2,2,2,1000}*. The mean value in the case mentioned is 2 if it wasn't for the `1000` at the end.
  
  * 1000 could be an outlier when you are trying to approximate the mean of the set with some algorithm. The algorithm is more likely to encounter a 2 in the unseen test set than the 1000 it encountered once in a training set.
  * When an algorithm like Linear Regressions "learns" something it is actually learning the weights and intercepts which modify the position of the line in the decision space.
  * The modifications are Translation(Additive operations) and Rotation(Multiplicative operations) to the said line. It is commonly referred to as the "weights" and "biases" respectively in case of a simple line Y = mx + c where m is the `slope` and c is the `intercept`.
  * The idea behind noise in the above text is that this:
    When an algorithm is trying to "learn" these weights you would want it to ignore the outliers - "Generalize" but you also want it to not ignore the feature data completely and introduce randomness - "Specialization".
  * How well the line is placed in your feature space is what determines your `Classification Effectiveness`.
  * In an ideal world you would want your decision boundary to ignore such outliers which are called `noise` in the above literature.
  
  # Pictures (because everyone likes them)
  
  * In the picture below the left image is what a good decision boundary is and the right image is what an overfitted Decision boundary is.
  * In the left-image case you are trading misclassifications at the cost of better generalizability(Consider that more likely you are going to see an x in the 2nd quadrant and the o was maybe an outlier). You do not fit your training set completely with a 100% accuracy.
    [![Normal Decision boundary vs Overfitted Decision Boundary](https://i.sstatic.net/LIBhh.png)](https://i.sstatic.net/LIBhh.png)
  
  # TL;DR
  
  You want the model to learn that the x(s) are on the right and the o(s) are on the left but not so specifically as to which x was where. The x in the 4th quadrant and the o in the second quadrant are `noisy factors`.
  
  
  ---
  
  