Body: <p>I am running a basic DQN (Deep Q-Network) on the Pong environment. Not a CNN, just a 3 layer linear neural net with ReLUs.</p>
  
  <p>It seems to work for the most part, but at some point, my model suffers from catastrophic performance loss:</p>
  
  <p><a href="https://i.sstatic.net/smEex.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/smEex.png" alt="Catastrophic forgetting"></a></p>
  
  <ol>
  <li><p>What is really the reason for that?</p></li>
  <li><p>What are the common ways to avoid this? Clipping the gradients? What else? </p></li>
  </ol>
  
  <p>(Reloading from previous successful checkpoints feels more like a hack, rather than a proper solution to this issue.)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-01T06:56:55.330Z
Comments.Text: Try to increase size of replay buffer or increase time update interval for target network.
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-01T20:48:15.777Z
Comments.Text: hmmm, tried both. It still happens. I did analyse the replays though, and it looks like basically a lot of the early strong gains come from learning a single trick to win. And then later the net tries to win without using that trick (i guess it forgets about that at some point), leading to that strong underperformance, until it learns that trick (or another) again. Overall stability improves eventually.
Comments.UserDisplayName: Muppet
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-02T14:22:49.330Z
Comments.Text: There was good paper from deep mind about similar phenomena - contradicting objectives causing plateau in learning -"Ray Interference: a Source of Plateaus in Deep Reinforcement Learning"
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-19T03:03:51.320Z
Comments.Text: @mirror2image That sounds like it could be a good starting point for an answer. Please consider writing one based around a short summary of that paper.
Comments.UserDisplayName: John Doucette
Tags: <reinforcement-learning><dqn><deep-rl>
Title: What could be causing the drastic performance drop of the DQN model on the Pong environment?
UserDisplayName: Muppet
ThreadText: # What could be causing the drastic performance drop of the DQN model on the Pong environment?
  
  (Asked by: username_0 on 2019-05-31)
  
  I am running a basic DQN (Deep Q-Network) on the Pong environment. Not a CNN, just a 3 layer linear neural net with ReLUs.
  
  It seems to work for the most part, but at some point, my model suffers from catastrophic performance loss:
  
  [![Catastrophic forgetting](https://i.sstatic.net/smEex.png)](https://i.sstatic.net/smEex.png)
  
  1. What is really the reason for that?
  2. What are the common ways to avoid this? Clipping the gradients? What else?
  
  (Reloading from previous successful checkpoints feels more like a hack, rather than a proper solution to this issue.)
  
  **Tags:** reinforcement-learning, dqn, deep-rl
  
  > **Comments:**
  > **username_1**: Try to increase size of replay buffer or increase time update interval for target network.  
  > **username_0**: hmmm, tried both. It still happens. I did analyse the replays though, and it looks like basically a lot of the early strong gains come from learning a single trick to win. And then later the net tries to win without using that trick (i guess it forgets about that at some point), leading to that strong underperformance, until it learns that trick (or another) again. Overall stability improves eventually.  
  > **username_1**: There was good paper from deep mind about similar phenomena - contradicting objectives causing plateau in learning -"Ray Interference: a Source of Plateaus in Deep Reinforcement Learning"  
  > **username_2**: @username_1 That sounds like it could be a good starting point for an answer. Please consider writing one based around a short summary of that paper.  
  
  ---
  
  ## 0 answers
  
  