Answers.Body: <p>Training scores improving (loss decreasing and accuracy increasing) whilst the opposite happens with cross validation and test data is a sign of <a href="https://en.wikipedia.org/wiki/Overfitting" rel="nofollow noreferrer">overfitting to the training data</a>. Your neural network is getting worse at generalising and no amount of further training will improve it - in fact the situation will get worse the more you train.</p>
  
  <p>This is the main reason you have CV data sets, to show you when this happens before you try to use your model against the test set or real world data. So it is not "peculiar" at all, but the CV set doing its job for you, allowing you notice something has gone wrong.</p>
  
  <p>To improve on the situation, you need to use some form of <a href="https://towardsdatascience.com/regularization-in-machine-learning-76441ddcf99a" rel="nofollow noreferrer">regularisation</a>. The simplest approach here would be to take your model from around 100 epochs (because it has best CV score that I can see). This is early stopping, and is a simple valid regularisation approach.</p>
  
  <p>Alternatives for neural networks include L2 weight regularisation (also called weight decay) and dropout.</p>
  
  <p>In addition to this, your question states that you only have 300 examples, and more features than examples. This is really tricky to generalise (and hard to tell if you have - for instance it looks like you only have 20 CV examples and 40 test examples, these numbers are very low and prone to giving you inaccurate estimates simply due to chance of what entries are in the data sets). I recommend you look into <a href="https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation" rel="nofollow noreferrer">K-fold cross-validation</a> in order to get more accurate scores, and help you choose the best model for generalisation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-30T03:16:47.283Z
Answers.Text: Thanks a lot. I've been suspecting the issue of overfitting, but had no idea why it really happen. Do you have any recommended resource on NN overfiting(AKA precision and bias problem)? I'm having a hard time analysing the result of the model..
Answers.UserDisplayName: Joshua Lee
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-30T08:58:53.947Z
Answers.Text: @JoshuaLee: I don't have any specific resource. Andrew Ng's course https://www.coursera.org/learn/machine-learning has some very useful guidance on best practices including the basic theory and techniques behind handling overfitting
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-30T23:48:49.020Z
Answers.Text: Thanks for your help. I do truly appreciate it.
Answers.UserDisplayName: Joshua Lee
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-28T09:17:24.363Z
Answers.LastActivityDate: 2019-06-28T09:17:24.363Z
Answers.UserDisplayName: Neil Slater
Body: <p>As mentioned in the title I'm using 300 Dataset example with 500 feature as an input.</p>
  
  <p>As I'm training the dataset, I found something peculiar.  Please look at the data shown below.</p>
  
  <blockquote>
    <p><strong>Iteration 5000 | Cost: 2.084241e-01</strong></p>
    
    <p>Training Set Accuracy: 100.000000</p>
    
    <p>CV Set Accuracy: 85.000000</p>
    
    <p>Test Set Accuracy: 97.500000</p>
    
    <p><strong>Iteration 3000 | Cost: 2.084241e-01</strong></p>
    
    <p>Training Set Accuracy: 98.958333</p>
    
    <p>CV Set Accuracy: 85.000000</p>
    
    <p>Test Set Accuracy: 97.500000</p>
    
    <p><strong>Iteration 1000 | Cost: 4.017322e-01</strong></p>
    
    <p>Training Set Accuracy: 96.875000</p>
    
    <p>CV Set Accuracy: 85.000000</p>
    
    <p>Test Set Accuracy: 97.500000</p>
    
    <p><strong>Iteration 500 | Cost: 5.515852e-01</strong></p>
    
    <p>Training Set Accuracy: 95.486111</p>
    
    <p>CV Set Accuracy: 90.000000</p>
    
    <p>Test Set Accuracy: 97.500000</p>
    
    <p><strong>Iteration 100 | Cost: 8.413299e-01</strong></p>
    
    <p>Training Set Accuracy: 90.625000</p>
    
    <p>CV Set Accuracy: 95.000000</p>
    
    <p>Test Set Accuracy: 97.500000</p>
    
    <p><strong>Iteration 50 | Cost: 8.483802e-01</strong></p>
    
    <p>Training Set Accuracy: 90.277778</p>
    
    <p>CV Set Accuracy: 95.000000</p>
    
    <p>Test Set Accuracy: 97.500000</p>
  </blockquote>
  
  <p>The trend is that as the Iteration(cost) increases(cost decreases), the training set accuracy increases as expected, but the CV Set/Test Set Accuracy decreases. My initial thought is that this has to do with precision/bias issue, but I really can't buy it. </p>
  
  <p>Anyone know what this entails? Or any reference? </p>
  
Tags: <neural-networks><machine-learning><gradient-descent>
Title: In NN, as iterations of Gradient descent increases, the accuracy of Test/CV set decreases. how can i resolve this?
UserDisplayName: Joshua Lee
ThreadText: # In NN, as iterations of Gradient descent increases, the accuracy of Test/CV set decreases. how can i resolve this?
  
  (Asked by: username_0 on 2019-06-28)
  
  As mentioned in the title I'm using 300 Dataset example with 500 feature as an input.
  
  As I'm training the dataset, I found something peculiar. Please look at the data shown below.
  
  > **Iteration 5000 | Cost: 2.084241e-01**
  >
  > Training Set Accuracy: 100.000000
  >
  > CV Set Accuracy: 85.000000
  >
  > Test Set Accuracy: 97.500000
  >
  > **Iteration 3000 | Cost: 2.084241e-01**
  >
  > Training Set Accuracy: 98.958333
  >
  > CV Set Accuracy: 85.000000
  >
  > Test Set Accuracy: 97.500000
  >
  > **Iteration 1000 | Cost: 4.017322e-01**
  >
  > Training Set Accuracy: 96.875000
  >
  > CV Set Accuracy: 85.000000
  >
  > Test Set Accuracy: 97.500000
  >
  > **Iteration 500 | Cost: 5.515852e-01**
  >
  > Training Set Accuracy: 95.486111
  >
  > CV Set Accuracy: 90.000000
  >
  > Test Set Accuracy: 97.500000
  >
  > **Iteration 100 | Cost: 8.413299e-01**
  >
  > Training Set Accuracy: 90.625000
  >
  > CV Set Accuracy: 95.000000
  >
  > Test Set Accuracy: 97.500000
  >
  > **Iteration 50 | Cost: 8.483802e-01**
  >
  > Training Set Accuracy: 90.277778
  >
  > CV Set Accuracy: 95.000000
  >
  > Test Set Accuracy: 97.500000
  
  The trend is that as the Iteration(cost) increases(cost decreases), the training set accuracy increases as expected, but the CV Set/Test Set Accuracy decreases. My initial thought is that this has to do with precision/bias issue, but I really can't buy it.
  
  Anyone know what this entails? Or any reference?
  
  **Tags:** neural-networks, machine-learning, gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-28. Score: 1 | Accepted answer
  
  Training scores improving (loss decreasing and accuracy increasing) whilst the opposite happens with cross validation and test data is a sign of [overfitting to the training data](https://en.wikipedia.org/wiki/Overfitting). Your neural network is getting worse at generalising and no amount of further training will improve it - in fact the situation will get worse the more you train.
  
  This is the main reason you have CV data sets, to show you when this happens before you try to use your model against the test set or real world data. So it is not "peculiar" at all, but the CV set doing its job for you, allowing you notice something has gone wrong.
  
  To improve on the situation, you need to use some form of [regularisation](https://towardsdatascience.com/regularization-in-machine-learning-76441ddcf99a). The simplest approach here would be to take your model from around 100 epochs (because it has best CV score that I can see). This is early stopping, and is a simple valid regularisation approach.
  
  Alternatives for neural networks include L2 weight regularisation (also called weight decay) and dropout.
  
  In addition to this, your question states that you only have 300 examples, and more features than examples. This is really tricky to generalise (and hard to tell if you have - for instance it looks like you only have 20 CV examples and 40 test examples, these numbers are very low and prone to giving you inaccurate estimates simply due to chance of what entries are in the data sets). I recommend you look into [K-fold cross-validation](https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation) in order to get more accurate scores, and help you choose the best model for generalisation.
  
  > **Comments:**
  > **username_0**: Thanks a lot. I've been suspecting the issue of overfitting, but had no idea why it really happen. Do you have any recommended resource on NN overfiting(AKA precision and bias problem)? I'm having a hard time analysing the result of the model..  
  > **username_1**: @username_0: I don't have any specific resource. Andrew Ng's course https://www.coursera.org/learn/machine-learning has some very useful guidance on best practices including the basic theory and techniques behind handling overfitting  
  > **username_0**: Thanks for your help. I do truly appreciate it.  
  
  ---
  
  