Answers.Body: <p>This is a common problem and does have a name. It is called "<a href="https://ourarchive.otago.ac.nz/handle/10523/1765" rel="nofollow noreferrer">catastrophic forgetting</a>" (link is just to a paper I found randomly when searching for the term).</p>
  
  <blockquote>
    <p>What could be causing this undesirable behavior? </p>
  </blockquote>
  
  <p>It happens only when using function approximation for value functions (e.g. a neural network trained to learn Q values), and is caused by the agent's own success. After a while the only samples that you will train with will be near-optimal high return cases, and a neural network will optimise the Q value approximation only for that recent data in order to get the best loss. This will usually mean much poorer predictions on unseen but different data, as the network overfits to states and actions only seen in the optimal policy. Eventually the agent will explore into an area where its predictions are way off. Then, because Q learning also uses its own predictions to <em>bootstrap</em> new Q values, this can start a runaway feedback process where the agent starts to choose a suboptimal path through the environment. </p>
  
  <p>Inside the hidden layers and weights, the neural network may have lost the ability to differentiate well between the states on the old, worse path and the newer better ones. It didn't need to differentiate, because it stopped needing to reduce loss on any data about the old states. So it will also start incorrectly associating the now poor results with the more optimal paths. It will behave at least partially as if the correct policy was set by the overfit Q predictions, but the values need adjusting - so as well as (correctly) reducing its value predictions of the suboptimal paths it has just encountered, it will also (incorrectly) reduce the value predictions of the optimal paths. </p>
  
  <p>Sometimes, the swing back to receiving lower returns during this process is so strong and/or incorrectly associated with the high return states along with the low ones, that the agent never properly recovers. Other times, the agent can re-learn fully and you get random oscillations over time between optimal and non-optimal agent behaviour.</p>
  
  <blockquote>
    <p>How can I combat it?</p>
  </blockquote>
  
  <p>This is still an ongoing area of research. Here are a couple of things that have worked for me:</p>
  
  <ul>
  <li><p>Low learning rates, and defences against sudden large gradients (e.g. gradient clipping).</p></li>
  <li><p>Regularisation. Sadly dropout seems not to work in RL, but weight decay is still useful to prevent over-fitting, and it also helps combat catastrophic forgetting because it prevents bootstrap estimates of long-unseen state/action combinations from returning radically different Q values to the rest of the system.</p></li>
  <li><p>Keep some early experience around from when the agent was still performing badly - this allows the agent to still train with some bad cases and prevents the Q function predicting that "everything is awesome" because it still has examples to learn from where this is not the case.</p>
  
  <ul>
  <li><p>For simple environments, such as inverted pendulum, just keeping some very early fully random behaviour in the experience replay table is enough. For instance if you have a table with 10000 observations (of <span class="math-container">$s, a, r, s'$</span>), keep 1000 of the first experiences in that table and don't discard them when the table is full. For more complex environments, this is not so useful, as the early random behaviour is too far removed from what the agent learns. </p></li>
  <li><p>The DQN "rainbow" paper uses <em>prioritized experience replay</em> to focus on areas where Q value predictions from the NN are not matching the observations. </p></li>
  </ul></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-23T00:29:30.463Z
Answers.LastActivityDate: 2019-02-23T00:44:14.250Z
Answers.LastEditDate: 2019-02-23T00:44:14.250Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have a reinforcement learning agent with both a positive and a negative terminal state.  After each episode during training, I am recording whether a success or failure occurred, and then I can compute a running ratio of success to failure.</p>
  
  <p>I am seeing a phenomenon where, at some point in time, my agent achieves a reasonably high success rate (~80%) for a 100-episode running average.  However, with further training, it seems to 'train itself out' of this behavior and ends the training sequence with a very low success rate (~10-20%).</p>
  
  <p>I am using an epsilon-greedy strategy whereby epsilon decays linearly from 1.0 to 0.1 for the first 10% of episodes and then remains at 0.1 for the remaining 90%.  As such, the 'training out' appears to occur some time where exploration only occurs with 10% probability.</p>
  
  <p>What could be causing this undesirable behavior? How can I combat it?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-22T17:08:22.323Z
Comments.Text: I don't get the connection between the expression "train itself out" and the fact that "ends the training sequence with a very low success rate" . Also, what exactly do you mean by "train itself out"?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-22T17:17:14.187Z
Comments.Text: At some point during training, it consistently achieves the goal with high success rate.  However, with more training, it seems to learn worse behavior.
Comments.UserDisplayName: Steven Davis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-22T18:58:33.583Z
Comments.Text: What kind of RL algorithm are you using? Q-learning, Sarsa, policy gradients, etc.? And are you using a tabular approach, or linear function approximation, or Deep RL, or something else?
Comments.UserDisplayName: Dennis Soemers
Tags: <reinforcement-learning><q-learning>
Title: What is happening when a reinforcement learning agent trains itself out of desired behavior?
UserDisplayName: Steven Davis
ThreadText: # What is happening when a reinforcement learning agent trains itself out of desired behavior?
  
  (Asked by: username_0 on 2019-02-22)
  
  I have a reinforcement learning agent with both a positive and a negative terminal state. After each episode during training, I am recording whether a success or failure occurred, and then I can compute a running ratio of success to failure.
  
  I am seeing a phenomenon where, at some point in time, my agent achieves a reasonably high success rate (~80%) for a 100-episode running average. However, with further training, it seems to 'train itself out' of this behavior and ends the training sequence with a very low success rate (~10-20%).
  
  I am using an epsilon-greedy strategy whereby epsilon decays linearly from 1.0 to 0.1 for the first 10% of episodes and then remains at 0.1 for the remaining 90%. As such, the 'training out' appears to occur some time where exploration only occurs with 10% probability.
  
  What could be causing this undesirable behavior? How can I combat it?
  
  **Tags:** reinforcement-learning, q-learning
  
  > **Comments:**
  > **username_1**: I don't get the connection between the expression "train itself out" and the fact that "ends the training sequence with a very low success rate" . Also, what exactly do you mean by "train itself out"?  
  > **username_0**: At some point during training, it consistently achieves the goal with high success rate.  However, with more training, it seems to learn worse behavior.  
  > **username_2**: What kind of RL algorithm are you using? Q-learning, Sarsa, policy gradients, etc.? And are you using a tabular approach, or linear function approximation, or Deep RL, or something else?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2019-02-23. Score: 4 | Accepted answer
  
  This is a common problem and does have a name. It is called "[catastrophic forgetting](https://ourarchive.otago.ac.nz/handle/10523/1765)" (link is just to a paper I found randomly when searching for the term).
  
  > What could be causing this undesirable behavior?
  
  It happens only when using function approximation for value functions (e.g. a neural network trained to learn Q values), and is caused by the agent's own success. After a while the only samples that you will train with will be near-optimal high return cases, and a neural network will optimise the Q value approximation only for that recent data in order to get the best loss. This will usually mean much poorer predictions on unseen but different data, as the network overfits to states and actions only seen in the optimal policy. Eventually the agent will explore into an area where its predictions are way off. Then, because Q learning also uses its own predictions to *bootstrap* new Q values, this can start a runaway feedback process where the agent starts to choose a suboptimal path through the environment.
  
  Inside the hidden layers and weights, the neural network may have lost the ability to differentiate well between the states on the old, worse path and the newer better ones. It didn't need to differentiate, because it stopped needing to reduce loss on any data about the old states. So it will also start incorrectly associating the now poor results with the more optimal paths. It will behave at least partially as if the correct policy was set by the overfit Q predictions, but the values need adjusting - so as well as (correctly) reducing its value predictions of the suboptimal paths it has just encountered, it will also (incorrectly) reduce the value predictions of the optimal paths.
  
  Sometimes, the swing back to receiving lower returns during this process is so strong and/or incorrectly associated with the high return states along with the low ones, that the agent never properly recovers. Other times, the agent can re-learn fully and you get random oscillations over time between optimal and non-optimal agent behaviour.
  
  > How can I combat it?
  
  This is still an ongoing area of research. Here are a couple of things that have worked for me:
  
  * Low learning rates, and defences against sudden large gradients (e.g. gradient clipping).
  * Regularisation. Sadly dropout seems not to work in RL, but weight decay is still useful to prevent over-fitting, and it also helps combat catastrophic forgetting because it prevents bootstrap estimates of long-unseen state/action combinations from returning radically different Q values to the rest of the system.
  * Keep some early experience around from when the agent was still performing badly - this allows the agent to still train with some bad cases and prevents the Q function predicting that "everything is awesome" because it still has examples to learn from where this is not the case.
  
    + For simple environments, such as inverted pendulum, just keeping some very early fully random behaviour in the experience replay table is enough. For instance if you have a table with 10000 observations (of $s, a, r, s'$), keep 1000 of the first experiences in that table and don't discard them when the table is full. For more complex environments, this is not so useful, as the early random behaviour is too far removed from what the agent learns.
    + The DQN "rainbow" paper uses *prioritized experience replay* to focus on areas where Q value predictions from the NN are not matching the observations.
  
  
  ---
  
  