Body: <p>I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some hyperparameter configurations this is followed by a rebound and again a slump. Here's what a typical reward plot looks like:<a href="https://i.sstatic.net/emnsw.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/emnsw.png" alt="1" /></a></p>
  <p>Since I'm new to reinforcement learning (this is my first shot at it), I don't know if this a common phenomenon. I know of catastrophic forgetting, but I believe that's not the case here, since this is more akin to a &quot;languishing dementia&quot;. As far as I understand, &quot;catastrophic forgetting&quot; is an abrupt event, which contrasts with a gradual change I've been seeing in my attempts.</p>
  <p>Is this some kind of general phenomenon with coverage in the existing literature or is this rather a quirk of my specific setup (algorithm + hyperparameters) for which the solution would be &quot;change the setup&quot;?</p>
  <p>For reference, the implementation I'm using: <a href="https://github.com/hirekk/pytorch-rl" rel="nofollow noreferrer">https://github.com/hirekk/pytorch-rl</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-28T12:27:07.803Z
Comments.Text: This could well be catastrophic forgetting. A few weeks ago, I came across a paper (I don't remember the name now, but if I find it I will post this comment as a answer below) that actually states that similar patterns of the performance can be due to catastrophic forgetting within the same environment, which is actually a multi-task environment.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-28T20:08:44.187Z
Comments.Text: @nbro Thank you very much, I wasn't aware that catastrophic forgetting can manifest that way. If this paper you mention managed to find its way back into your hands, I'd appreciate greatly you pointing me to it.
Comments.UserDisplayName: Hirek Kubica
Tags: <reinforcement-learning><reference-request><deep-rl><ddpg><catastrophic-forgetting>
Title: Gradual decrease in performance of a DDPG agent
UserDisplayName: Hirek Kubica
ThreadText: # Gradual decrease in performance of a DDPG agent
  
  (Asked by: username_0 on 2021-01-27)
  
  I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some hyperparameter configurations this is followed by a rebound and again a slump. Here's what a typical reward plot looks like:[![1](https://i.sstatic.net/emnsw.png)](https://i.sstatic.net/emnsw.png)
  
  Since I'm new to reinforcement learning (this is my first shot at it), I don't know if this a common phenomenon. I know of catastrophic forgetting, but I believe that's not the case here, since this is more akin to a "languishing dementia". As far as I understand, "catastrophic forgetting" is an abrupt event, which contrasts with a gradual change I've been seeing in my attempts.
  
  Is this some kind of general phenomenon with coverage in the existing literature or is this rather a quirk of my specific setup (algorithm + hyperparameters) for which the solution would be "change the setup"?
  
  For reference, the implementation I'm using: <https://github.com/hirekk/pytorch-rl>
  
  **Tags:** reinforcement-learning, reference-request, deep-rl, ddpg, catastrophic-forgetting
  > **Comments:**
  > **username_1**: This could well be catastrophic forgetting. A few weeks ago, I came across a paper (I don't remember the name now, but if I find it I will post this comment as a answer below) that actually states that similar patterns of the performance can be due to catastrophic forgetting within the same environment, which is actually a multi-task environment.  
  > **username_0**: @nbro Thank you very much, I wasn't aware that catastrophic forgetting can manifest that way. If this paper you mention managed to find its way back into your hands, I'd appreciate greatly you pointing me to it.  
  
  ---
  
  ## 0 answers
  
  