Answers.Body: <p>The validation loss settles exactly at an error of one. Probably means there's something off with either the kind of data validation set has or with something in the training. An exact validation loss of one almost definitely means there's something off. </p>
  
  <p>I'd recommend before doing anything thoroughly go through your data or see if there's anything to debug in the model itself. Considering training error reduces there's probably something different about either the formatting of the validation data or the validation data itself. </p>
  
  <p>A mild description of the type of data and exact problem in hand could further help.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-29T23:42:46.003Z
Answers.Text: What is 'almost definitely' wrong with an MSE of 1, and why?
Answers.UserDisplayName: desertnaut
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-30T07:05:19.940Z
Answers.Text: Why would validation error trends settle at exactly one when it can take any real value. The only way validation error would settle is if the training defined a minima too hard for it to come out of exactly at a validation error of 1, meaning there's surely something to debug in the data.
Answers.UserDisplayName: ashenoy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-30T10:28:04.727Z
Answers.Text: You seem to overinterpret a coarse plot which, if zoomed in, would most probably reveal fluctuations around 1.0. It would be safer to ask for a clarification here than assuming that it is *exactly* 1.0. The plot by itself shows nothing wrong
Answers.UserDisplayName: desertnaut
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-28T09:55:36.703Z
Answers.LastActivityDate: 2019-11-28T09:55:36.703Z
Answers.UserDisplayName: ashenoy
Answers.Body: <p>Depends on what does 1 represent in your task. 
  If you are trying to predict household prices and 1 represents \$1, I think the average validation loss is good. If 1 represents \$10000 in this case, probably something is not right. But remember that there are 2 parts contributing to the overall loss. The mse loss and the l2 penalty loss. (Also remember that most optimizers already implement l2 penalty as weight decay. So you do not need to add it separately)</p>
  
  <p>Some suggestions.</p>
  
  <ol>
  <li><p>Check if your data has any outliers/ anomalies. Based on your task you should know as to what you can do with these data points. Also see if your dataset has high variance. </p></li>
  <li><p>If you are worried about over-fitting, think about your data again. Less data + more parameters often leads to over-fitting. If your dataset is too small, you need to think again.  </p></li>
  <li><p>Try to adjust the number of hidden units and observe the results.  </p></li>
  <li><p>Try using cross validation.  </p></li>
  <li><p>Alternatively, try using different optimizers and see what happens (Try Adam). </p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-27T04:13:31.617Z
Answers.LastActivityDate: 2019-11-27T04:13:31.617Z
Answers.UserDisplayName: keshik
Answers.Body: <p>The telltale <strong>signature</strong> of overfitting is when your validation loss starts increasing, while your training loss continues decreasing, i.e.:</p>
  
  <p><a href="https://i.sstatic.net/KLM4h.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KLM4h.png" alt="adapted from Wikipedia"></a></p>
  
  <p>(Image adapted from Wikipedia entry on <a href="https://en.wikipedia.org/wiki/Overfitting" rel="nofollow noreferrer">overfitting</a>)</p>
  
  <p>It is clear that this does not happen in your diagram, hence your model does not overfit.</p>
  
  <p>A difference between a training and a validation score by itself does <strong>not</strong> signify overfitting. This is just the <strong>generalization gap</strong>, i.e. the <em>expected</em> gap in the performance between the training and validation sets; quoting from a recent <a href="https://ai.googleblog.com/2019/07/predicting-generalization-gap-in-deep.html" rel="nofollow noreferrer">blog post by Google AI</a>:</p>
  
  <blockquote>
    <p>An important concept for understanding generalization is the <em>generalization gap</em>, i.e., the difference between a model’s performance on training data and its performance on unseen data drawn from the same distribution.</p>
  </blockquote>
  
  <p>An MSE of 1.0 (or any other specific value, for that matter) by itself cannot be considered "good" or "bad"; everything depends on the context, i.e. of the particular problem and the actual magnitude of your dependent variable: if you are trying to predict something that is in the order of some thousands (or even hundreds), an MSE of 1.0 does not sound bad; it's not the same if your dependent variable takes values, say, in [0, 1] or similar.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-30T19:37:16.313Z
Answers.Text: The output is two values in floating point between -30 and 30: what can you say about MSE 1.0?
Answers.UserDisplayName: Andrea Favilli
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-30T00:05:25.293Z
Answers.LastActivityDate: 2019-11-30T00:05:25.293Z
Answers.UserDisplayName: desertnaut
Body: <p>Loss is MSE; orange is validation loss, blue training loss. The task is NN regression (18 inputs, 2 outputs), one layer 300 hidden units.
  <a href="https://i.sstatic.net/Rs1sq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Rs1sq.png" alt=""></a></p>
  
  <p>Tuning the lr, mom, l2 regularization parameters this is the best validation loss I can obtain. Can be considered overfitting? Is 1 a bad vl loss value for a regression task?</p>
  
Tags: <neural-networks><machine-learning><objective-functions><learning-algorithms><cross-validation>
Title: How do you interpret this learning curve?
UserDisplayName: Andrea Favilli
ThreadText: # How do you interpret this learning curve?
  
  (Asked by: username_0 on 2019-11-27)
  
  Loss is MSE; orange is validation loss, blue training loss. The task is NN regression (18 inputs, 2 outputs), one layer 300 hidden units.
  [![](https://i.sstatic.net/Rs1sq.png)](https://i.sstatic.net/Rs1sq.png)
  
  Tuning the lr, mom, l2 regularization parameters this is the best validation loss I can obtain. Can be considered overfitting? Is 1 a bad vl loss value for a regression task?
  
  **Tags:** neural-networks, machine-learning, objective-functions, learning-algorithms, cross-validation
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-11-28. Score: 3
  
  The validation loss settles exactly at an error of one. Probably means there's something off with either the kind of data validation set has or with something in the training. An exact validation loss of one almost definitely means there's something off.
  
  I'd recommend before doing anything thoroughly go through your data or see if there's anything to debug in the model itself. Considering training error reduces there's probably something different about either the formatting of the validation data or the validation data itself.
  
  A mild description of the type of data and exact problem in hand could further help.
  
  > **Comments:**
  > **username_2**: What is 'almost definitely' wrong with an MSE of 1, and why?  
  > **username_1**: Why would validation error trends settle at exactly one when it can take any real value. The only way validation error would settle is if the training defined a minima too hard for it to come out of exactly at a validation error of 1, meaning there's surely something to debug in the data.  
  > **username_2**: You seem to overinterpret a coarse plot which, if zoomed in, would most probably reveal fluctuations around 1.0. It would be safer to ask for a clarification here than assuming that it is *exactly* 1.0. The plot by itself shows nothing wrong  
  
  ---
  
  ### Answer by username_3 on 2019-11-27. Score: 3
  
  Depends on what does 1 represent in your task.
  If you are trying to predict household prices and 1 represents \$1, I think the average validation loss is good. If 1 represents \$10000 in this case, probably something is not right. But remember that there are 2 parts contributing to the overall loss. The mse loss and the l2 penalty loss. (Also remember that most optimizers already implement l2 penalty as weight decay. So you do not need to add it separately)
  
  Some suggestions.
  
  1. Check if your data has any outliers/ anomalies. Based on your task you should know as to what you can do with these data points. Also see if your dataset has high variance.
  2. If you are worried about over-fitting, think about your data again. Less data + more parameters often leads to over-fitting. If your dataset is too small, you need to think again.
  3. Try to adjust the number of hidden units and observe the results.
  4. Try using cross validation.
  5. Alternatively, try using different optimizers and see what happens (Try Adam).
  
  
  ---
  
  ### Answer by username_2 on 2019-11-30. Score: 1
  
  The telltale **signature** of overfitting is when your validation loss starts increasing, while your training loss continues decreasing, i.e.:
  
  [![adapted from Wikipedia](https://i.sstatic.net/KLM4h.png)](https://i.sstatic.net/KLM4h.png)
  
  (Image adapted from Wikipedia entry on [overfitting](https://en.wikipedia.org/wiki/Overfitting))
  
  It is clear that this does not happen in your diagram, hence your model does not overfit.
  
  A difference between a training and a validation score by itself does **not** signify overfitting. This is just the **generalization gap**, i.e. the *expected* gap in the performance between the training and validation sets; quoting from a recent [blog post by Google AI](https://ai.googleblog.com/2019/07/predicting-generalization-gap-in-deep.html):
  
  > An important concept for understanding generalization is the *generalization gap*, i.e., the difference between a model’s performance on training data and its performance on unseen data drawn from the same distribution.
  
  An MSE of 1.0 (or any other specific value, for that matter) by itself cannot be considered "good" or "bad"; everything depends on the context, i.e. of the particular problem and the actual magnitude of your dependent variable: if you are trying to predict something that is in the order of some thousands (or even hundreds), an MSE of 1.0 does not sound bad; it's not the same if your dependent variable takes values, say, in [0, 1] or similar.
  
  > **Comments:**
  > **username_0**: The output is two values in floating point between -30 and 30: what can you say about MSE 1.0?  
  
  ---
  
  