Answers.Body: <p>It is not 100% clear, but this seems like an instance of <a href="https://en.wikipedia.org/wiki/Catastrophic_interference" rel="noreferrer">catastrophic forgetting</a>. This is something that often impacts reinforcement learning.</p>
  <p>I have answered <a href="https://datascience.stackexchange.com/questions/56053/why-could-my-ddqn-get-significantly-worse-after-beating-the-game-repeatedly/56055#56055">a very similar question on Data Science stack exchange</a>, and reproduce the same answer here.</p>
  <hr />
  <p>This is called &quot;catastrophic forgetting&quot; and can be a serious problem in many RL scenarios.</p>
  <p>If you trained a neural network to recognise cats and dogs and did the following:</p>
  <ul>
  <li><p>Train it for many epochs on a full dataset until you got a high accuracy.</p>
  </li>
  <li><p>Continue to train it, but remove all the cat pictures.</p>
  </li>
  </ul>
  <p>Then in a relatively short space of time, the NN would start to lose accuracy. It would forget what a cat looks like. It would learn that its task was to switch the dog prediction as high as possible, just because on average everything in the training population was a dog.</p>
  <p>Something very similar happens in your DQN experience replay memory. Once it gets good at a task, it may only experience success. Eventually, only successful examples are in its memory. The NN forgets what failure looks like (what the states are, and what it should predict for their values), and predicts high values for everything.</p>
  <p>Later on, when something bad happens and the NNs high predicted value is completely wrong, the error can be high. In addition the NN may have incorrectly &quot;linked&quot; features of the state representation so that it cannot distinguish which parts of the feature space are the cause of this. This creates odd effects in terms of what it learns about values of all states. Often the NN will behave incorrectly for a few episodes but then re-learn optimal behaviour. But it is also possible that it completely breaks and never recovers.</p>
  <p>There is lots of active research into catastrophic forgetting and I suggest you search that term to find out some of the many types of mitigation you could use.</p>
  <p>For Cartpole, I found a very simple hack made the learning very stable. Keep aside some percentage of replay memory stocked with the initial poor performing random exploration. Reserving say 10% to this long term memory is enough to make learning in Cartpole rock solid, as the NN always has a few examples of what not to do. The idea unfortunately does not scale well to more complex environments, but it is a nice demonstration. For a more sophisticated look at similar solutions you could see the paper &quot;<a href="https://en.wikipedia.org/wiki/Catastrophic_interference" rel="noreferrer">The importance of experience replay database composition in deep reinforcement learning</a>&quot;</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-19T22:47:32.217Z
Answers.Text: Great explanation !
Answers.UserDisplayName: Jose Mar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-15T01:19:15.900Z
Answers.Text: This answer is fantastic and I know its been a couple of years but I wonder if leaving the final exploration rate >0 (say 0.05 or so) is sufficient to counteract this? It should mean that the agent will always be sampling some random actions (of which there will inevitably be things the agent shouldn't do)
Answers.UserDisplayName: Jarym
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-15T09:09:59.233Z
Answers.Text: @Jarym: It might help to tweak this as a hyperparameter. However, catastrophic forgetting is often observed in DQN-based learning systems where there already is a mimimum exploration rate, so it is not a fix. Setting a high exploration rate can compromise overall earning rate and eventual optimality, especially in harder problems with long episodes.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-15T09:13:33.387Z
Answers.Text: @Jarym: Different exploration *schemes*, as opposed to tweaking the exploration rate, could help. For instance, an approach like DynaQ+, where exploration rates increase over time and tend to encourage large, multi-step explorations, or coherent noise approaches which tend to explore or exploit over multiple timesteps. Both these approaches will sample data further away from current successful trajectories on a routine basis
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-29T07:42:50.993Z
Answers.LastActivityDate: 2020-09-29T08:47:42.563Z
Answers.LastEditDate: 2020-09-29T08:47:42.563Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually increases during training and then drastically drops. In the simulator, this corresponds to the drone exhibiting cool collision avoidance after a few thousand episodes. However, after training for more iterations it starts taking <em>counterintuitive</em> actions such as simply crashing into a wall (I have checked to ensure that there is no exploration at play over here).</p>
  <p>Does this have to do with overfitting? I am unable to understand why my rewards are falling this way.</p>
  
Tags: <reinforcement-learning><q-learning><dqn><rewards>
Title: Why do my rewards reduce after extensive training using D3QN?
UserDisplayName: desert_ranger
ThreadText: # Why do my rewards reduce after extensive training using D3QN?
  
  (Asked by: username_0 on 2020-09-28)
  
  I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually increases during training and then drastically drops. In the simulator, this corresponds to the drone exhibiting cool collision avoidance after a few thousand episodes. However, after training for more iterations it starts taking *counterintuitive* actions such as simply crashing into a wall (I have checked to ensure that there is no exploration at play over here).
  
  Does this have to do with overfitting? I am unable to understand why my rewards are falling this way.
  
  **Tags:** reinforcement-learning, q-learning, dqn, rewards
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-29. Score: 7 | Accepted answer
  
  It is not 100% clear, but this seems like an instance of [catastrophic forgetting](https://en.wikipedia.org/wiki/Catastrophic_interference). This is something that often impacts reinforcement learning.
  
  I have answered [a very similar question on Data Science stack exchange](https://datascience.stackexchange.com/questions/56053/why-could-my-ddqn-get-significantly-worse-after-beating-the-game-repeatedly/56055#56055), and reproduce the same answer here.
  
  ---
  
  This is called "catastrophic forgetting" and can be a serious problem in many RL scenarios.
  
  If you trained a neural network to recognise cats and dogs and did the following:
  
  * Train it for many epochs on a full dataset until you got a high accuracy.
  * Continue to train it, but remove all the cat pictures.
  
  Then in a relatively short space of time, the NN would start to lose accuracy. It would forget what a cat looks like. It would learn that its task was to switch the dog prediction as high as possible, just because on average everything in the training population was a dog.
  
  Something very similar happens in your DQN experience replay memory. Once it gets good at a task, it may only experience success. Eventually, only successful examples are in its memory. The NN forgets what failure looks like (what the states are, and what it should predict for their values), and predicts high values for everything.
  
  Later on, when something bad happens and the NNs high predicted value is completely wrong, the error can be high. In addition the NN may have incorrectly "linked" features of the state representation so that it cannot distinguish which parts of the feature space are the cause of this. This creates odd effects in terms of what it learns about values of all states. Often the NN will behave incorrectly for a few episodes but then re-learn optimal behaviour. But it is also possible that it completely breaks and never recovers.
  
  There is lots of active research into catastrophic forgetting and I suggest you search that term to find out some of the many types of mitigation you could use.
  
  For Cartpole, I found a very simple hack made the learning very stable. Keep aside some percentage of replay memory stocked with the initial poor performing random exploration. Reserving say 10% to this long term memory is enough to make learning in Cartpole rock solid, as the NN always has a few examples of what not to do. The idea unfortunately does not scale well to more complex environments, but it is a nice demonstration. For a more sophisticated look at similar solutions you could see the paper "[The importance of experience replay database composition in deep reinforcement learning](https://en.wikipedia.org/wiki/Catastrophic_interference)"
  
  > **Comments:**
  > **username_2**: Great explanation !  
  > **username_3**: This answer is fantastic and I know its been a couple of years but I wonder if leaving the final exploration rate >0 (say 0.05 or so) is sufficient to counteract this? It should mean that the agent will always be sampling some random actions (of which there will inevitably be things the agent shouldn't do)  
  > **username_1**: @Jarym: It might help to tweak this as a hyperparameter. However, catastrophic forgetting is often observed in DQN-based learning systems where there already is a mimimum exploration rate, so it is not a fix. Setting a high exploration rate can compromise overall earning rate and eventual optimality, especially in harder problems with long episodes.  
  > **username_1**: @Jarym: Different exploration *schemes*, as opposed to tweaking the exploration rate, could help. For instance, an approach like DynaQ+, where exploration rates increase over time and tend to encourage large, multi-step explorations, or coherent noise approaches which tend to explore or exploit over multiple timesteps. Both these approaches will sample data further away from current successful trajectories on a routine basis  
  
  ---
  
  