Body: <p>I am following a course on machine learning and am confused about the bias-variance trade-off relationship
  to learning curves in classification.</p>
  <p>I am seeing some conflicting information online on this.</p>
  <p>The scikit-learn learning curve looks like the top 2 curves here: <a href="https://i.sstatic.net/2LmCL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2LmCL.png" alt="Learning curve" /></a><br />
  <sub>(source: <a href="https://scikit-learn.org/stable/_images/sphx_glr_plot_learning_curve_0011.png" rel="nofollow noreferrer">scikit-learn.org</a>)</sub></p>
  <p>What I don't understand is: how do we read bias from this? If we look at this image
  where each blue dot is a model. I think the bias would be the green curve being high. But high bias
  indicates underfitting, right? So shouldn't the red curve be high then too?</p>
  <p><img src="https://external-content.duckduckgo.com/iu/?u=https://tse4.mm.bing.net/th?id=OIP.-pwwSpcPJcxzDaRH40w95QAAAA&amp;pid=Api&amp;f=1" alt="Bias variance tradeoff" /></p>
  <p>High variance would be the gap between green and red, is this correct?</p>
  <p>My question is how do the red and green curves relate to underfitting and overfitting,
  and how do learning curves fit with the figure with the concentric circles? Is bias purely related to the red curve, or is a model with a low validation score and high train score also a high bias model?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-14T01:51:29.083Z
Comments.Text: Where did you take these pictures from? Can you also clarify what the _score_ and *fit_times* (on the y-axis) are? I assume that the "cross-validation score" is the model evaluated on separate data. From the plots, it seems that you evaluate and train the data on the same number of samples. Overfitting typically happens when the performance of the model increases on the training data, but the performance decreases on the validation data. Under-fitting occurs when the model does not even perform well on the training data.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-14T01:56:55.990Z
Comments.Text: See https://en.wikipedia.org/wiki/Overfitting and https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff. If I interpret correctly your first two plots on the top, the right one should correspond to no _apparent_ over-fitting and underfitting. The left one could indicate that there's some underfitting, because the performance decreases on the training data, even though it increases on the validation data, but you want your model also to perform well on the training data. I don't know how to interpret the other plots because I don't know what the x and y axes are supposed to represent.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-14T01:59:29.023Z
Comments.Text: Moreover, please, ask only one question per post. Here, you're asking many questions, and it's difficult to address all of them in detail. If you have many questions, ask one for each post. Given that I am not able to properly answer all your question, this really means that your post requires more details and that you should actually split it into multiple posts, one for each distinct question.
Comments.UserDisplayName: nbro
Tags: <bias-variance-tradeoff><learning-curve>
Title: Bias-variance tradeoff and learning curves for non-deep learning models
UserDisplayName: ML-Student-1996
ThreadText: # Bias-variance tradeoff and learning curves for non-deep learning models
  
  (Asked by: username_0 on 2020-11-12)
  
  I am following a course on machine learning and am confused about the bias-variance trade-off relationship
  to learning curves in classification.
  
  I am seeing some conflicting information online on this.
  
  The scikit-learn learning curve looks like the top 2 curves here: [![Learning curve](https://i.sstatic.net/2LmCL.png)](https://i.sstatic.net/2LmCL.png)  
  (source: [scikit-learn.org](https://scikit-learn.org/stable/_images/sphx_glr_plot_learning_curve_0011.png))
  
  What I don't understand is: how do we read bias from this? If we look at this image
  where each blue dot is a model. I think the bias would be the green curve being high. But high bias
  indicates underfitting, right? So shouldn't the red curve be high then too?
  
  ![Bias variance tradeoff](https://external-content.duckduckgo.com/iu/?u=https://tse4.mm.bing.net/th?id=OIP.-pwwSpcPJcxzDaRH40w95QAAAA&pid=Api&f=1)
  
  High variance would be the gap between green and red, is this correct?
  
  My question is how do the red and green curves relate to underfitting and overfitting,
  and how do learning curves fit with the figure with the concentric circles? Is bias purely related to the red curve, or is a model with a low validation score and high train score also a high bias model?
  
  **Tags:** bias-variance-tradeoff, learning-curve
  > **Comments:**
  > **username_1**: Where did you take these pictures from? Can you also clarify what the _score_ and *fit_times* (on the y-axis) are? I assume that the "cross-validation score" is the model evaluated on separate data. From the plots, it seems that you evaluate and train the data on the same number of samples. Overfitting typically happens when the performance of the model increases on the training data, but the performance decreases on the validation data. Under-fitting occurs when the model does not even perform well on the training data.  
  > **username_1**: See https://en.wikipedia.org/wiki/Overfitting and https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff. If I interpret correctly your first two plots on the top, the right one should correspond to no _apparent_ over-fitting and underfitting. The left one could indicate that there's some underfitting, because the performance decreases on the training data, even though it increases on the validation data, but you want your model also to perform well on the training data. I don't know how to interpret the other plots because I don't know what the x and y axes are supposed to represent.  
  > **username_1**: Moreover, please, ask only one question per post. Here, you're asking many questions, and it's difficult to address all of them in detail. If you have many questions, ask one for each post. Given that I am not able to properly answer all your question, this really means that your post requires more details and that you should actually split it into multiple posts, one for each distinct question.  
  
  ---
  
  ## 0 answers
  
  