Answers.Body: <p>Additional features can also cause overfitting if they have low or misleading information.</p>
  
  <p>Consider the following problem:</p>
  
  <p><span class="math-container">$X = [1, 3, 3, 4, 5]$</span>, <span class="math-container">$Y = [1, 3, 4, 4, 5]$</span>.</p>
  
  <p>Suppose that the real dataset was generated from the relationship:</p>
  
  <p><span class="math-container">$Y = X$</span>, with a probability of 0.2 of adding or subtracting 1.</p>
  
  <p>A reasonable model estimate is <span class="math-container">$Y = X$</span>. <strong>Note</strong> that <strong>no model can fit this data perfectly</strong>, because the two 3 inputs map to different outputs.</p>
  
  <p>Now, suppose we add a new feature: a random number between <span class="math-container">$0$</span> and <span class="math-container">$10$</span>: 
  <span class="math-container">$W = [1 ,5, 2, 6, 3]$</span></p>
  
  <p>It may not be obvious, but a sufficiently deep and broad neural network can learn a new function: </p>
  
  <p><span class="math-container">$g(W) = 1$</span> if <span class="math-container">$W = 2,4,7,8,9,$</span> or <span class="math-container">$0$</span>.</p>
  
  <p><span class="math-container">$g(W) = 0$</span> otherwise.</p>
  
  <p>and define a new prediction: <span class="math-container">$Y = X - g(W)$</span>.</p>
  
  <p>This happens to produce a <em>perfect</em> fit on the training data. However, it will perform extremely poorly on new data (like a test set), because it has learned a meaningless pattern out of random noise. Coincidentally, it will be wrong on about 50% of samples, while our first model will be wrong on only 20% of samples.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-22T23:27:22.263Z
Answers.Text: Maybe you mean Y = [1, 2, 3, 4, 5]?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-22T23:30:28.410Z
Answers.Text: @nbro No, I intended for there to be some error in the original set, but on second look, I need to add some extra data for this example to make sense.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-22T21:58:13.180Z
Answers.LastActivityDate: 2020-03-21T00:53:18.397Z
Answers.LastEditDate: 2020-03-21T00:53:18.397Z
Answers.UserDisplayName: John Doucette
Body: <p>I have a Remaining Useful Life (RUL) prediction problem that I want to solve. When I added two or more features as inputs to my ANN, the accuracy of my ANN has been decreased. More precisely, I've added features like RMS or KURTOSIS (or both). I was expecting the system to improve, but it is getting worse. </p>
  
  <p>Why might this be happening? What are the potential reasons for this degradation in performance?</p>
  
  <p>I know that when we added more nodes in layers (like hidden layers), overfitting can happen. Would that be related to my problem: using more than two features?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-22T20:14:53.993Z
Comments.Text: A simple case is if the new features noisy.
Comments.UserDisplayName: SpiderRico
Tags: <neural-networks><machine-learning><prediction><overfitting><feature-selection>
Title: How come that the addition of features can decrease the performance of a neural network?
UserDisplayName: Aref.a
ThreadText: # How come that the addition of features can decrease the performance of a neural network?
  
  (Asked by: username_0 on 2017-12-08)
  
  I have a Remaining Useful Life (RUL) prediction problem that I want to solve. When I added two or more features as inputs to my ANN, the accuracy of my ANN has been decreased. More precisely, I've added features like RMS or KURTOSIS (or both). I was expecting the system to improve, but it is getting worse.
  
  Why might this be happening? What are the potential reasons for this degradation in performance?
  
  I know that when we added more nodes in layers (like hidden layers), overfitting can happen. Would that be related to my problem: using more than two features?
  
  **Tags:** neural-networks, machine-learning, prediction, overfitting, feature-selection
  > **Comments:**
  > **username_1**: A simple case is if the new features noisy.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-10-22. Score: 3
  
  Additional features can also cause overfitting if they have low or misleading information.
  
  Consider the following problem:
  
  $X = [1, 3, 3, 4, 5]$, $Y = [1, 3, 4, 4, 5]$.
  
  Suppose that the real dataset was generated from the relationship:
  
  $Y = X$, with a probability of 0.2 of adding or subtracting 1.
  
  A reasonable model estimate is $Y = X$. **Note** that **no model can fit this data perfectly**, because the two 3 inputs map to different outputs.
  
  Now, suppose we add a new feature: a random number between $0$ and $10$:
  $W = [1 ,5, 2, 6, 3]$
  
  It may not be obvious, but a sufficiently deep and broad neural network can learn a new function:
  
  $g(W) = 1$ if $W = 2,4,7,8,9,$ or $0$.
  
  $g(W) = 0$ otherwise.
  
  and define a new prediction: $Y = X - g(W)$.
  
  This happens to produce a *perfect* fit on the training data. However, it will perform extremely poorly on new data (like a test set), because it has learned a meaningless pattern out of random noise. Coincidentally, it will be wrong on about 50% of samples, while our first model will be wrong on only 20% of samples.
  
  > **Comments:**
  > **username_3**: Maybe you mean Y = [1, 2, 3, 4, 5]?  
  > **username_2**: @nbro No, I intended for there to be some error in the original set, but on second look, I need to add some extra data for this example to make sense.  
  
  ---
  
  