Answers.Body: <p>The right plot is about <a href="https://www.pnas.org/doi/full/10.1073/pnas.1903070116" rel="nofollow noreferrer">Deep Double Descent</a>, a <em>phenomenon</em> observed in Deep Learning that challenges the classical belief (left plot) in statistical learning theory.</p>
  <ul>
  <li>The first half of the right plot, depicts the classical empirical risk minimization situation in which you seek for the optimal model capacity: the one that balances bias and variance, achieving a low training error and good generalization.</li>
  <li>It has been observed that if you keep increasing the capacity of some model (assuming proper regularization) your <strong>training error goes to zero</strong> (the <em>interpolation threshold</em>) and, unexpectedly, the generalization (test) error decreases too as the capacity is increased!</li>
  <li>This modern interpolation regime were over-parameterized models actually generalize much better than models with the just-right capacity contradicts the classical view and the bias-variance trade-off.</li>
  </ul>
  <p>One explanation for this is that, for DL models variance is not only in the sampling of the data used for training (like assumed by classical ML), but also in the weight initialization, optimization, and training procedure as well. Seems that over-parameterization <em>decreases</em> such variance allowing for better generalization.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-16T18:56:37.667Z
Answers.Text: Got it, that makes sense. How does over-parameterization solve the variance from weight initialization, optimization, and training procedure?
Answers.UserDisplayName: 9j09jf02jsd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-20T16:33:57.763Z
Answers.Text: One assumption can be that over-parameterized models have many weights, and some of them are redundant, also there could be symmetries among them. These can help either with the optimization and training (since some weights may already point in the right direction of improvement), and variance from weight init (redundancy may limit the variance of initialization.)
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-20T22:11:51.543Z
Answers.Text: Hmm I'm not fully convinced, I thought the right "direction" of improvement is specified by the negative gradient - I don't see how more parameters helps this. And regarding weight init variance, I thought having more weights could create more local minima that SGD could get stuck in. I don't see how in general, more parameters => less variance
Answers.UserDisplayName: 9j09jf02jsd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-15T17:48:17.480Z
Answers.LastActivityDate: 2023-05-15T17:48:17.480Z
Answers.UserDisplayName: Luca Anzalone
Body: <p><a href="https://i.sstatic.net/dEVwz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/dEVwz.png" alt="" /></a></p>
  <p>From Berkeley CS182, SP22: <a href="https://cs182sp22.github.io/assets/lecture_slides/2022.01.26-ml-review-pt2.pdf" rel="nofollow noreferrer">https://cs182sp22.github.io/assets/lecture_slides/2022.01.26-ml-review-pt2.pdf</a>.</p>
  <p>Can someone help me interpret this diagram? I understand the graph on the left, but I don't understand how in the right graph, the test risk starts going back down. I'm unfamiliar with the &quot;interpolating regime&quot;, so maybe that would explain some things.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-18T21:57:12.173Z
Comments.Text: A good interactive explanation can be seen at MLU-explain: https://mlu-explain.github.io/bias-variance/
Comments.UserDisplayName: shamisen
Tags: <machine-learning><deep-learning><overfitting><bias-variance-tradeoff>
Title: How does deep learning overcome overfitting?
UserDisplayName: 9j09jf02jsd
ThreadText: # How does deep learning overcome overfitting?
  
  (Asked by: username_0 on 2023-05-14)
  
  [![](https://i.sstatic.net/dEVwz.png)](https://i.sstatic.net/dEVwz.png)
  
  From Berkeley CS182, SP22: <https://cs182sp22.github.io/assets/lecture_slides/2022.01.26-ml-review-pt2.pdf>.
  
  Can someone help me interpret this diagram? I understand the graph on the left, but I don't understand how in the right graph, the test risk starts going back down. I'm unfamiliar with the "interpolating regime", so maybe that would explain some things.
  
  **Tags:** machine-learning, deep-learning, overfitting, bias-variance-tradeoff
  > **Comments:**
  > **username_1**: A good interactive explanation can be seen at MLU-explain: https://mlu-explain.github.io/bias-variance/  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-15. Score: 3
  
  The right plot is about [Deep Double Descent](https://www.pnas.org/doi/full/10.1073/pnas.1903070116), a *phenomenon* observed in Deep Learning that challenges the classical belief (left plot) in statistical learning theory.
  
  * The first half of the right plot, depicts the classical empirical risk minimization situation in which you seek for the optimal model capacity: the one that balances bias and variance, achieving a low training error and good generalization.
  * It has been observed that if you keep increasing the capacity of some model (assuming proper regularization) your **training error goes to zero** (the *interpolation threshold*) and, unexpectedly, the generalization (test) error decreases too as the capacity is increased!
  * This modern interpolation regime were over-parameterized models actually generalize much better than models with the just-right capacity contradicts the classical view and the bias-variance trade-off.
  
  One explanation for this is that, for DL models variance is not only in the sampling of the data used for training (like assumed by classical ML), but also in the weight initialization, optimization, and training procedure as well. Seems that over-parameterization *decreases* such variance allowing for better generalization.
  
  > **Comments:**
  > **username_0**: Got it, that makes sense. How does over-parameterization solve the variance from weight initialization, optimization, and training procedure?  
  > **username_2**: One assumption can be that over-parameterized models have many weights, and some of them are redundant, also there could be symmetries among them. These can help either with the optimization and training (since some weights may already point in the right direction of improvement), and variance from weight init (redundancy may limit the variance of initialization.)  
  > **username_0**: Hmm I'm not fully convinced, I thought the right "direction" of improvement is specified by the negative gradient - I don't see how more parameters helps this. And regarding weight init variance, I thought having more weights could create more local minima that SGD could get stuck in. I don't see how in general, more parameters => less variance  
  
  ---
  
  