<p>Additional features can also cause overfitting if they have low or misleading information.</p>

<p>Consider the following problem:</p>

<p><span class="math-container">$X = [1, 3, 3, 4, 5]$</span>, <span class="math-container">$Y = [1, 3, 4, 4, 5]$</span>.</p>

<p>Suppose that the real dataset was generated from the relationship:</p>

<p><span class="math-container">$Y = X$</span>, with a probability of 0.2 of adding or subtracting 1.</p>

<p>A reasonable model estimate is <span class="math-container">$Y = X$</span>. <strong>Note</strong> that <strong>no model can fit this data perfectly</strong>, because the two 3 inputs map to different outputs.</p>

<p>Now, suppose we add a new feature: a random number between <span class="math-container">$0$</span> and <span class="math-container">$10$</span>: 
<span class="math-container">$W = [1 ,5, 2, 6, 3]$</span></p>

<p>It may not be obvious, but a sufficiently deep and broad neural network can learn a new function: </p>

<p><span class="math-container">$g(W) = 1$</span> if <span class="math-container">$W = 2,4,7,8,9,$</span> or <span class="math-container">$0$</span>.</p>

<p><span class="math-container">$g(W) = 0$</span> otherwise.</p>

<p>and define a new prediction: <span class="math-container">$Y = X - g(W)$</span>.</p>

<p>This happens to produce a <em>perfect</em> fit on the training data. However, it will perform extremely poorly on new data (like a test set), because it has learned a meaningless pattern out of random noise. Coincidentally, it will be wrong on about 50% of samples, while our first model will be wrong on only 20% of samples.</p>

CC BY-SA 4.0
2019-10-22T23:27:22.263Z
Maybe you mean Y = [1, 2, 3, 4, 5]?
nbro
CC BY-SA 4.0
2019-10-22T23:30:28.410Z
@nbro No, I intended for there to be some error in the original set, but on second look, I need to add some extra data for this example to make sense.
John Doucette
CC BY-SA 4.0
2019-10-22T21:58:13.180Z
2020-03-21T00:53:18.397Z
2020-03-21T00:53:18.397Z
John Doucette
<p>I have a Remaining Useful Life (RUL) prediction problem that I want to solve. When I added two or more features as inputs to my ANN, the accuracy of my ANN has been decreased. More precisely, I've added features like RMS or KURTOSIS (or both). I was expecting the system to improve, but it is getting worse. </p>

<p>Why might this be happening? What are the potential reasons for this degradation in performance?</p>

<p>I know that when we added more nodes in layers (like hidden layers), overfitting can happen. Would that be related to my problem: using more than two features?</p>

CC BY-SA 4.0
2020-03-22T20:14:53.993Z
A simple case is if the new features noisy.
SpiderRico
<neural-networks><machine-learning><prediction><overfitting><feature-selection>
How come that the addition of features can decrease the performance of a neural network?
Aref.a
# How come that the addition of features can decrease the performance of a neural network?

(Asked by: username_0 on 2017-12-08)

I have a Remaining Useful Life (RUL) prediction problem that I want to solve. When I added two or more features as inputs to my ANN, the accuracy of my ANN has been decreased. More precisely, I've added features like RMS or KURTOSIS (or both). I was expecting the system to improve, but it is getting worse.

Why might this be happening? What are the potential reasons for this degradation in performance?

I know that when we added more nodes in layers (like hidden layers), overfitting can happen. Would that be related to my problem: using more than two features?

**Tags:** neural-networks, machine-learning, prediction, overfitting, feature-selection

> **Comments:**
> **username_1**: A simple case is if the new features noisy.  

---

## 1 answers

### Answer by username_2 on 2019-10-22. Score: 3

Additional features can also cause overfitting if they have low or misleading information.

Consider the following problem:

$X = [1, 3, 3, 4, 5]$, $Y = [1, 3, 4, 4, 5]$.

Suppose that the real dataset was generated from the relationship:

$Y = X$, with a probability of 0.2 of adding or subtracting 1.

A reasonable model estimate is $Y = X$. **Note** that **no model can fit this data perfectly**, because the two 3 inputs map to different outputs.

Now, suppose we add a new feature: a random number between $0$ and $10$:
$W = [1 ,5, 2, 6, 3]$

It may not be obvious, but a sufficiently deep and broad neural network can learn a new function:

$g(W) = 1$ if $W = 2,4,7,8,9,$ or $0$.

$g(W) = 0$ otherwise.

and define a new prediction: $Y = X - g(W)$.

This happens to produce a *perfect* fit on the training data. However, it will perform extremely poorly on new data (like a test set), because it has learned a meaningless pattern out of random noise. Coincidentally, it will be wrong on about 50% of samples, while our first model will be wrong on only 20% of samples.

> **Comments:**
> **username_3**: Maybe you mean Y = [1, 2, 3, 4, 5]?  
> **username_2**: @username_3 No, I intended for there to be some error in the original set, but on second look, I need to add some extra data for this example to make sense.  

---

