Answers.Body: <p>It doesn't tell you very much, to be honest. It <em>does</em> mean that (assuming your training and validation distributions are similar) your model could get the same results on your validation set should you train on that, but that would still be overfitting.</p>
  
  <p>Really, the only useful thing overfitting tells you is that you don't have enough regularisation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-10T08:44:19.893Z
Answers.Text: overfitting tells you that your model don't generalize well. Regularization is not the only method to reduce overfitting (think of a bigger dataset)
Answers.UserDisplayName: Jérémy Blain
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-10T08:46:48.587Z
Answers.Text: I would argue that using more data, augmenting data, reducing the size of the net, etc. are all forms of regularisation.
Answers.UserDisplayName: Omegastick
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-10T08:32:38.880Z
Answers.LastActivityDate: 2018-10-10T08:32:38.880Z
Answers.UserDisplayName: Omegastick
Body: <p>As shown below, my deep neural network is overfitting :
  <a href="https://i.sstatic.net/AfDBu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AfDBu.png" alt="Cross-entropy loss"></a>
  <a href="https://i.sstatic.net/janOU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/janOU.png" alt="Accuracy"></a></p>
  
  <p><em>where the <strong>blue</strong> lines is the metrics obtained with <strong>training</strong> set and <strong>red</strong> lines with <strong>validation</strong> set</em></p>
  
  <p><strong>Is there anything I can infer from the fact that the accuracy on the training sets is really high</strong> (almost 1) <strong>?</strong></p>
  
  <p>From what I understand, it means that the complexity of my model is enough / too big. But does it means my model could theoretically reach such a score on validation set with same dataset and appropriate hyperparameters ? With same hyperparameters but bigger dataset ? </p>
  
  <p>My question is <strong>not</strong> how to avoid overfitting.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-10T08:37:52.057Z
Comments.Text: "But does it means my model could theoretically reach such a score on validation set with same dataset and appropriate hyperparameters ? With same hyperparameters but bigger dataset ?" It's a question about how to avoid overfitting ;)
Comments.UserDisplayName: Jérémy Blain
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-11T06:14:14.057Z
Comments.Text: @JérémyBlain Sorry for my poor english, I'm not native. My sentence might not convey the sens I wished. I was asking some interpretation based on the (overfitting) training accuracy. I know the methods to avoid overfitting. I was more asking about experience with overfitting, if it is possible to reach such a good accuracy without applying all the counter-overfitting methods.
Comments.UserDisplayName: Astariul
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-13T08:57:41.753Z
Comments.Text: See chapter 2.5 of [Analysis and Optimization of Convolutional Neural Network Architectures](https://arxiv.org/abs/1707.09725)
Comments.UserDisplayName: Martin Thoma
Tags: <neural-networks><machine-learning><deep-learning><overfitting>
Title: Interpretation of a good overfitting score
UserDisplayName: Astariul
ThreadText: # Interpretation of a good overfitting score
  
  (Asked by: username_0 on 2018-10-10)
  
  As shown below, my deep neural network is overfitting :
  [![Cross-entropy loss](https://i.sstatic.net/AfDBu.png)](https://i.sstatic.net/AfDBu.png)
  [![Accuracy](https://i.sstatic.net/janOU.png)](https://i.sstatic.net/janOU.png)
  
  *where the **blue** lines is the metrics obtained with **training** set and **red** lines with **validation** set*
  
  **Is there anything I can infer from the fact that the accuracy on the training sets is really high** (almost 1) **?**
  
  From what I understand, it means that the complexity of my model is enough / too big. But does it means my model could theoretically reach such a score on validation set with same dataset and appropriate hyperparameters ? With same hyperparameters but bigger dataset ?
  
  My question is **not** how to avoid overfitting.
  
  **Tags:** neural-networks, machine-learning, deep-learning, overfitting
  > **Comments:**
  > **username_1**: "But does it means my model could theoretically reach such a score on validation set with same dataset and appropriate hyperparameters ? With same hyperparameters but bigger dataset ?" It's a question about how to avoid overfitting ;)  
  > **username_0**: @JérémyBlain Sorry for my poor english, I'm not native. My sentence might not convey the sens I wished. I was asking some interpretation based on the (overfitting) training accuracy. I know the methods to avoid overfitting. I was more asking about experience with overfitting, if it is possible to reach such a good accuracy without applying all the counter-overfitting methods.  
  > **username_2**: See chapter 2.5 of [Analysis and Optimization of Convolutional Neural Network Architectures](https://arxiv.org/abs/1707.09725)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2018-10-10. Score: 0 | Accepted answer
  
  It doesn't tell you very much, to be honest. It *does* mean that (assuming your training and validation distributions are similar) your model could get the same results on your validation set should you train on that, but that would still be overfitting.
  
  Really, the only useful thing overfitting tells you is that you don't have enough regularisation.
  
  > **Comments:**
  > **username_1**: overfitting tells you that your model don't generalize well. Regularization is not the only method to reduce overfitting (think of a bigger dataset)  
  > **username_3**: I would argue that using more data, augmenting data, reducing the size of the net, etc. are all forms of regularisation.  
  
  ---
  
  