Answers.Body: <p><strong>The Problem of Overfitting</strong></p>
  <p>In most cases, when you increase a lot the number of epochs your model finally overfits. This is because your model reaches the point that it does not learn anymore but tries to remember what it has seen before. This is overfitting. So there is often a trade-off between the number of epochs and overfitting. In general, a good way to avoid overfitting, except for fine-tuning, regularization, dropout, etc, is to understand what you have from the learning curve. In most cases, overfitting happens, after some epochs have passed, and, as a result, the training error still decreases, whereas the validation error fluctuates or increases. If so, you should save only the learning updates before overfitting appears and/or validation error is minimum.<br />
  Methods: early-stopping, checkpoint.
  useful link: <a href="https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/" rel="nofollow noreferrer">https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-11T23:36:07.290Z
Answers.LastActivityDate: 2022-05-29T04:52:16.730Z
Answers.LastEditDate: 2022-05-29T04:52:16.730Z
Answers.UserDisplayName: ddaedalus
Body: <p>I've seen some comments in online articles/tutorials or <a href="https://stackoverflow.com/questions/53942612/why-too-many-epochs-will-cause-overfitting">Stack Overflow questions</a> which suggest that increasing the number of epochs can result in overfitting. But my intuition tells me that there should be no direct relationship at all between the number of epochs and overfitting. <strong>So I'm looking for an answer which explains if I'm right or wrong (or whatever's in between).</strong></p>
  <p>Here's my reasoning though. To overfit, you need to have enough free parameters (I think this is called &quot;capacity&quot; in neural networks) in your model to generate a function that can replicate the sample data points. If you don't have enough free parameters, you'll never overfit. You might just underfit.</p>
  <p>So really, if you don't have too many free parameters, you could run infinite epochs and never overfit. If you have too many free parameters, then yes, the more epochs you have the more likely it is that you get to a place where you're overfitting. But that's just because running more epochs revealed the root cause: too many free parameters. The real loss function doesn't care about how many epochs you run. It existed the moment you defined your model structure before you ever even tried to do gradient descent on it.</p>
  <p>In fact, I'd venture as far as to say: assuming you have the computational resources and time, you should always aim to run as many epochs as possible because that will tell you whether your model is prone to overfitting. Your best model will be the one that provides great training and validation accuracy, no matter how many epochs you run it for.</p>
  <p><strong>EDIT</strong>
  While reading more into this, I realise I forgot to take into account that you can arbitrarily vary the sample size as well. Given a fixed model, a smaller sample size is more prone to overfitting. And then that kind of makes me doubt my intuition above. Still happy to get an answer though!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-28T15:51:27.773Z
Comments.Text: @DuttaA haha well thanks. I'm about to post an edit which makes me feel not so definitely correct. Check it out in a minute
Comments.UserDisplayName: Alexander Soare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-28T16:27:26.213Z
Comments.Text: I think it also depends on how you train the network. If you have some kind of regulariser (i.e. you add noise to the loss) while training the net, you may not overfit.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-29T06:39:52.577Z
Comments.Text: Early stopping of training (before loss converge) can sometimes act as stochastic regularizer and  decrease overfitting. That method  is highly unreliable though and could cause "hyperparameter overfitting". That is relationships between number of epochs and overfitting.
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-05T09:57:34.370Z
Comments.Text: If my answer was helpful, please check it as right answer.
Comments.UserDisplayName: ddaedalus
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-05T10:57:02.943Z
Comments.Text: @ddaedalus sorry but I don't think your answer addresses the question. I think it explains what overfitting is and why it's a problem (as you aptly titled it). My question takes that knowledge as a prior and tries to challenge what I believe to be a common misconception.
Comments.UserDisplayName: Alexander Soare
Tags: <neural-networks><gradient-descent><overfitting><capacity>
Title: Is running more epochs really a direct cause of overfitting?
UserDisplayName: Alexander Soare
ThreadText: # Is running more epochs really a direct cause of overfitting?
  
  (Asked by: username_0 on 2019-12-28)
  
  I've seen some comments in online articles/tutorials or [Stack Overflow questions](https://stackoverflow.com/questions/53942612/why-too-many-epochs-will-cause-overfitting) which suggest that increasing the number of epochs can result in overfitting. But my intuition tells me that there should be no direct relationship at all between the number of epochs and overfitting. **So I'm looking for an answer which explains if I'm right or wrong (or whatever's in between).**
  
  Here's my reasoning though. To overfit, you need to have enough free parameters (I think this is called "capacity" in neural networks) in your model to generate a function that can replicate the sample data points. If you don't have enough free parameters, you'll never overfit. You might just underfit.
  
  So really, if you don't have too many free parameters, you could run infinite epochs and never overfit. If you have too many free parameters, then yes, the more epochs you have the more likely it is that you get to a place where you're overfitting. But that's just because running more epochs revealed the root cause: too many free parameters. The real loss function doesn't care about how many epochs you run. It existed the moment you defined your model structure before you ever even tried to do gradient descent on it.
  
  In fact, I'd venture as far as to say: assuming you have the computational resources and time, you should always aim to run as many epochs as possible because that will tell you whether your model is prone to overfitting. Your best model will be the one that provides great training and validation accuracy, no matter how many epochs you run it for.
  
  **EDIT**
  While reading more into this, I realise I forgot to take into account that you can arbitrarily vary the sample size as well. Given a fixed model, a smaller sample size is more prone to overfitting. And then that kind of makes me doubt my intuition above. Still happy to get an answer though!
  
  **Tags:** neural-networks, gradient-descent, overfitting, capacity
  
  > **Comments:**
  > **username_0**: @DuttaA haha well thanks. I'm about to post an edit which makes me feel not so definitely correct. Check it out in a minute  
  > **username_1**: I think it also depends on how you train the network. If you have some kind of regulariser (i.e. you add noise to the loss) while training the net, you may not overfit.  
  > **username_2**: Early stopping of training (before loss converge) can sometimes act as stochastic regularizer and  decrease overfitting. That method  is highly unreliable though and could cause "hyperparameter overfitting". That is relationships between number of epochs and overfitting.  
  > **username_3**: If my answer was helpful, please check it as right answer.  
  > **username_0**: @username_3 sorry but I don't think your answer addresses the question. I think it explains what overfitting is and why it's a problem (as you aptly titled it). My question takes that knowledge as a prior and tries to challenge what I believe to be a common misconception.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-05-11. Score: 3
  
  **The Problem of Overfitting**
  
  In most cases, when you increase a lot the number of epochs your model finally overfits. This is because your model reaches the point that it does not learn anymore but tries to remember what it has seen before. This is overfitting. So there is often a trade-off between the number of epochs and overfitting. In general, a good way to avoid overfitting, except for fine-tuning, regularization, dropout, etc, is to understand what you have from the learning curve. In most cases, overfitting happens, after some epochs have passed, and, as a result, the training error still decreases, whereas the validation error fluctuates or increases. If so, you should save only the learning updates before overfitting appears and/or validation error is minimum.  
  Methods: early-stopping, checkpoint.
  useful link: <https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/>
  
  
  ---
  
  