Body: <p>Given an ill-looking reinforcement learning performance curve, what are the possible causes?</p>
  <p>For example, in the graph below, y-axis is <code>return</code>, and x-axis is training <code>epoch</code>, the algorithm (PPO in this case) is able to learn quite well at the beginning (increasing return), but somehow it starts to &quot;unlearn&quot; afterwards...</p>
  <p><a href="https://i.sstatic.net/w1jF0zY8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/w1jF0zY8.png" alt="learning-curve" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-24T11:18:35.677Z
Comments.Text: I would guess this curve is for PPO during training, and not separate test/validation runs with tweaks to exploration parameters? That's worth making clear because expected behaviour for many RL algorithms will differ between training and test runs, but a lot of people don't bother with separate automated testing during training (not saying you should, and I don't know enough about PPO to know if it would help)
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-24T11:25:38.787Z
Comments.Text: Do you have any details of the environment that you can share? That may help. An upper bound on the return could also help, and some indication of how stochastic the environment is
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-27T22:02:41.070Z
Comments.Text: Please, provide more details about the curves. You even occluded the legend, for some mysterious reason. Maybe this is a homework problem?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deep-rl>
Title: Given an ill-looking reinforcement learning performance curve, what are the possible causes?
UserDisplayName: em1971
ThreadText: # Given an ill-looking reinforcement learning performance curve, what are the possible causes?
  
  (Asked by: username_0 on 2024-10-23)
  
  Given an ill-looking reinforcement learning performance curve, what are the possible causes?
  
  For example, in the graph below, y-axis is `return`, and x-axis is training `epoch`, the algorithm (PPO in this case) is able to learn quite well at the beginning (increasing return), but somehow it starts to "unlearn" afterwards...
  
  [![learning-curve](https://i.sstatic.net/w1jF0zY8.png)](https://i.sstatic.net/w1jF0zY8.png)
  
  **Tags:** reinforcement-learning, deep-rl
  > **Comments:**
  > **username_1**: I would guess this curve is for PPO during training, and not separate test/validation runs with tweaks to exploration parameters? That's worth making clear because expected behaviour for many RL algorithms will differ between training and test runs, but a lot of people don't bother with separate automated testing during training (not saying you should, and I don't know enough about PPO to know if it would help)  
  > **username_1**: Do you have any details of the environment that you can share? That may help. An upper bound on the return could also help, and some indication of how stochastic the environment is  
  > **username_2**: Please, provide more details about the curves. You even occluded the legend, for some mysterious reason. Maybe this is a homework problem?  
  
  ---
  
  ## 0 answers
  
  