Answers.Body: <p>Could it be due to <strong>catastrophic forgetting/interference</strong>? If once the agent reaches 320K steps it becomes good at the task, it might start to experience only success. This could cause the agent to forget what failure looks like.</p>
  <p>Not sure how to handle such problem in PPO, but I suggest you search about it and you take a look at this <a href="https://arxiv.org/abs/2006.05990" rel="nofollow noreferrer">paper</a>, which provide lots of insights and recommendations for PPO.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-23T13:46:17.777Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-23T11:28:44.140Z
Answers.LastActivityDate: 2022-05-23T11:28:44.140Z
Answers.UserDisplayName: Albert Rial
Body: <p>I have trained an RL agent (PPO) for 6 million steps to solve the OpenAI gym LunarLander-v2. Surprisingly, the agent performs best already after 320K steps and is getting worse after that.
  In the tensorboard log, I can see that the mean, min reward and explained variance do have the highest values at 320k training steps.</p>
  <p><a href="https://i.sstatic.net/JqsT8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JqsT8.png" alt="enter image description here" /></a></p>
  <p>I have seen this with stable-baselines and rllib and with other environments as well.</p>
  <p>I am wondering why this is the case. Is that a normal behaviour in reinforcement learning? Or do I have to modify some training parameters to continue improving the RL agent?</p>
  <p>I would like to see that the agent is increasing the min, mean reward so that it reaches almost the max reward. Is that realistic?</p>
  
Tags: <reinforcement-learning><rewards><proximal-policy-optimization><gym><lunar-lander>
Title: Why is training longer not better in reinforcement learning?
UserDisplayName: Martin S
ThreadText: # Why is training longer not better in reinforcement learning?
  
  (Asked by: username_0 on 2022-05-21)
  
  I have trained an RL agent (PPO) for 6 million steps to solve the OpenAI gym LunarLander-v2. Surprisingly, the agent performs best already after 320K steps and is getting worse after that.
  In the tensorboard log, I can see that the mean, min reward and explained variance do have the highest values at 320k training steps.
  
  [![enter image description here](https://i.sstatic.net/JqsT8.png)](https://i.sstatic.net/JqsT8.png)
  
  I have seen this with stable-baselines and rllib and with other environments as well.
  
  I am wondering why this is the case. Is that a normal behaviour in reinforcement learning? Or do I have to modify some training parameters to continue improving the RL agent?
  
  I would like to see that the agent is increasing the min, mean reward so that it reaches almost the max reward. Is that realistic?
  
  **Tags:** reinforcement-learning, rewards, proximal-policy-optimization, gym, lunar-lander
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-23. Score: 2
  
  Could it be due to **catastrophic forgetting/interference**? If once the agent reaches 320K steps it becomes good at the task, it might start to experience only success. This could cause the agent to forget what failure looks like.
  
  Not sure how to handle such problem in PPO, but I suggest you search about it and you take a look at this [paper](https://arxiv.org/abs/2006.05990), which provide lots of insights and recommendations for PPO.
  
  > **Comments:**
  > **username_2**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  