Body: <p>It is mentioned by <a href="https://arxiv.org/abs/1902.10250" rel="nofollow noreferrer">Fu 2019</a> that overfitting might have a negative effect on training DQN. They showed that with either <em>early stopping</em> or <em>experience replay</em> this effect could be reduced. The first is reducing overfitting, the latter is increasing data.</p>
  
  <p>It doesn't only have negative effects on the returns though, my test shows that it has a negative effect on <em>value errors</em> as well (diff. between predict V and ground truth V). I observed frequently with limited data that the training diverged almost 100% of the time (on small nets). Since increasing the amount of data could reduce the chance of divergence, I think this is an effect from overfitting. </p>
  
  <p>Overfitting should mean low training loss, however, my observation is that there is a strong correlation between <em>TD loss</em> and <em>value error</em>. That is if I see a jump in TD loss, I could expect to see a jump in value error around that moment. </p>
  
  <p>Or it is not overfitting because it is not really <em>fit</em> (i.e. high loss) but over-optimization that is for sure. </p>
  
  <p>Now the question is why? </p>
  
  <p>There are two points:</p>
  
  <ul>
  <li>If it is overfitting, overfitting should have a positive effect because remembering values for all training states correctly is hardly a bad thing. (In fact, my training data is a <em>superset</em> of my testing data, so remembering should be fine.)</li>
  <li>If it doesn't fit, this begs a question what over-optimization really does. It doesn't seem to fit, but it does have a negative effect. How could that be?</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-30T15:44:33.697Z
Comments.Text: Is your training data really a superset of the test data? If so, that defeats the purpose of having the test data - you should never include your test data in your training set! Otherwise, you have no unbiased means of determining whether you're overfitting or not.
Comments.UserDisplayName: Nuclear Hoagie
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-01T02:54:11.920Z
Comments.Text: @NuclearWang I only use the test data to calculate *value error*. Anyways, overfitting or not could not be seen by looking at the value error since the optimization is done in *TD loss*. I expect to see *low TD loss*. Even in this training data, TD loss is also increasing contradicting to a general idea of overfitting. 
  
  I should point out that increasing TD loss could be a side effect of *divergence*.
Comments.UserDisplayName: Phizaz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-01T12:54:36.650Z
Comments.Text: Then what do you mean by superset? Simply memorizing the training set is usually a bad thing, since there's no generalization to unseen data. But you seem to be saying that it's not a problem because your training data points are part of the test set. Low value error on the training set doesn't necessarily indicate overfitting, that's only the case if you also see high value error on the test set. If value error on the train and test is similar then the model is not overfit, and that can happen with a both a low-error (good) and a high-error (poor) model.
Comments.UserDisplayName: Nuclear Hoagie
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-02T15:26:20.467Z
Comments.Text: @NuclearWang You seem to suggest that I cannot say the word "overfit" because I don't have a proper test data. You are right on that. I kinda assume that to be the case because I do train *looooong* enough. Saying it is overfitting due to loss is hard in RL with bootstrapping because I always see divergence *before* the end of training, not having a chance to see loss coverge to zero. I now hypothesize that maybe *overfitting causes divergence*. That's why overfitting is bad in RL. Not in a sense that generalization is bad in general.
Comments.UserDisplayName: Phizaz
Tags: <reinforcement-learning><deep-rl>
Title: Why overfitting is bad in DQN?
UserDisplayName: Phizaz
ThreadText: # Why overfitting is bad in DQN?
  
  (Asked by: username_0 on 2019-04-30)
  
  It is mentioned by [Fu 2019](https://arxiv.org/abs/1902.10250) that overfitting might have a negative effect on training DQN. They showed that with either *early stopping* or *experience replay* this effect could be reduced. The first is reducing overfitting, the latter is increasing data.
  
  It doesn't only have negative effects on the returns though, my test shows that it has a negative effect on *value errors* as well (diff. between predict V and ground truth V). I observed frequently with limited data that the training diverged almost 100% of the time (on small nets). Since increasing the amount of data could reduce the chance of divergence, I think this is an effect from overfitting.
  
  Overfitting should mean low training loss, however, my observation is that there is a strong correlation between *TD loss* and *value error*. That is if I see a jump in TD loss, I could expect to see a jump in value error around that moment.
  
  Or it is not overfitting because it is not really *fit* (i.e. high loss) but over-optimization that is for sure.
  
  Now the question is why?
  
  There are two points:
  
  * If it is overfitting, overfitting should have a positive effect because remembering values for all training states correctly is hardly a bad thing. (In fact, my training data is a *superset* of my testing data, so remembering should be fine.)
  * If it doesn't fit, this begs a question what over-optimization really does. It doesn't seem to fit, but it does have a negative effect. How could that be?
  
  **Tags:** reinforcement-learning, deep-rl
  > **Comments:**
  > **username_1**: Is your training data really a superset of the test data? If so, that defeats the purpose of having the test data - you should never include your test data in your training set! Otherwise, you have no unbiased means of determining whether you're overfitting or not.  
  > **username_0**: @NuclearWang I only use the test data to calculate *value error*. Anyways, overfitting or not could not be seen by looking at the value error since the optimization is done in *TD loss*. I expect to see *low TD loss*. Even in this training data, TD loss is also increasing contradicting to a general idea of overfitting.   I should point out that increasing TD loss could be a side effect of *divergence*.  
  > **username_1**: Then what do you mean by superset? Simply memorizing the training set is usually a bad thing, since there's no generalization to unseen data. But you seem to be saying that it's not a problem because your training data points are part of the test set. Low value error on the training set doesn't necessarily indicate overfitting, that's only the case if you also see high value error on the test set. If value error on the train and test is similar then the model is not overfit, and that can happen with a both a low-error (good) and a high-error (poor) model.  
  > **username_0**: @NuclearWang You seem to suggest that I cannot say the word "overfit" because I don't have a proper test data. You are right on that. I kinda assume that to be the case because I do train *looooong* enough. Saying it is overfitting due to loss is hard in RL with bootstrapping because I always see divergence *before* the end of training, not having a chance to see loss coverge to zero. I now hypothesize that maybe *overfitting causes divergence*. That's why overfitting is bad in RL. Not in a sense that generalization is bad in general.  
  
  ---
  
  ## 0 answers
  
  