Answers.Body: <p>This question is related to some unresolved issues in reinforcement learning with approximation.</p>
  <p>The following is a partial answer, but is the advice you should initially follow, unless you have a clear goal for doing otherwise:</p>
  <p><em>A neural network approximation function, like the Q network in DQN, is sensitive to the distribution of examples. For the best accuracy metrics (lowest loss) on test and unseen data the samples should be learned with the same distribution as they will be seen when using the learned function. So the replay memory should not have its distribution modified.</em></p>
  <p>Provided the <strong>samples</strong> from the replay memory follow a distribution close to the real distribution, then your loss objective will work effectively to produce accurate value estimates. So you could compress the results, provided you weighted the sampling to match what was observed.</p>
  <p>However, it gets more complex than this in practice:</p>
  <ul>
  <li>Off-policy samples are <em>not</em> in the same distribution as the target policy anyway. This is one reason you should use a relatively low <span class="math-container">$\epsilon$</span> when using Q learning with approximation.</li>
  <li>Further to that, <em>only</em> training with the exact correct distribution from the optimal policy can risk <a href="https://en.wikipedia.org/wiki/Catastrophic_interference" rel="nofollow noreferrer">catastrophic forgetting</a>, where the network loses generalisation to any other policy choices, and can more easily make mistakes because it cannot tell when choices are bad.</li>
  </ul>
  <p>There are a few different approaches to address catastrophic forgetting in reinforcement learning, but the basic message is that the &quot;correct&quot; balance of training data to most usefully train a Q function approximation is not fully resolved. This won't really impact your decision to de-duplicate the states - you probably shouldn't, or rather it won't be worth the effort. It does mean it could be worth experimenting a little with what data does go into the memory, and how it is sampled. I had some success on simple problems by keeping 10% of the memory for &quot;long-term&quot; memory of poorer policies, to stabilise the approximation so it can predict what the bad decisions are regardless of what the most recent policies have sampled.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-27T17:47:55.283Z
Answers.Text: It's so funny to read how they say that the state distribution should be as close as possible, and anybody with some RL notion considers DQN as "off policy"
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-27T13:12:42.963Z
Answers.LastActivityDate: 2024-05-27T13:12:42.963Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm training an RL agent/model (DRL/DQN).</p>
  <p>Say that, for each learning step, the memory replay used by the agent to learn, has N elements (experiences) stored, where only X are unique elements (experiences), meaning X &lt;= N.</p>
  <p>Would it be beneficial for the agent to sample from the whole memory or from a memory with unique elements?</p>
  
Tags: <reinforcement-learning><deep-rl><q-learning><dqn><experience-replay>
Title: Should the experience replay memory only contain unique experiences?
UserDisplayName: Jose Alberto Salazar
ThreadText: # Should the experience replay memory only contain unique experiences?
  
  (Asked by: username_0 on 2024-05-27)
  
  I'm training an RL agent/model (DRL/DQN).
  
  Say that, for each learning step, the memory replay used by the agent to learn, has N elements (experiences) stored, where only X are unique elements (experiences), meaning X <= N.
  
  Would it be beneficial for the agent to sample from the whole memory or from a memory with unique elements?
  
  **Tags:** reinforcement-learning, deep-rl, q-learning, dqn, experience-replay
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-27. Score: 1
  
  This question is related to some unresolved issues in reinforcement learning with approximation.
  
  The following is a partial answer, but is the advice you should initially follow, unless you have a clear goal for doing otherwise:
  
  *A neural network approximation function, like the Q network in DQN, is sensitive to the distribution of examples. For the best accuracy metrics (lowest loss) on test and unseen data the samples should be learned with the same distribution as they will be seen when using the learned function. So the replay memory should not have its distribution modified.*
  
  Provided the **samples** from the replay memory follow a distribution close to the real distribution, then your loss objective will work effectively to produce accurate value estimates. So you could compress the results, provided you weighted the sampling to match what was observed.
  
  However, it gets more complex than this in practice:
  
  * Off-policy samples are *not* in the same distribution as the target policy anyway. This is one reason you should use a relatively low $\epsilon$ when using Q learning with approximation.
  * Further to that, *only* training with the exact correct distribution from the optimal policy can risk [catastrophic forgetting](https://en.wikipedia.org/wiki/Catastrophic_interference), where the network loses generalisation to any other policy choices, and can more easily make mistakes because it cannot tell when choices are bad.
  
  There are a few different approaches to address catastrophic forgetting in reinforcement learning, but the basic message is that the "correct" balance of training data to most usefully train a Q function approximation is not fully resolved. This won't really impact your decision to de-duplicate the states - you probably shouldn't, or rather it won't be worth the effort. It does mean it could be worth experimenting a little with what data does go into the memory, and how it is sampled. I had some success on simple problems by keeping 10% of the memory for "long-term" memory of poorer policies, to stabilise the approximation so it can predict what the bad decisions are regardless of what the most recent policies have sampled.
  
  > **Comments:**
  > **username_2**: It's so funny to read how they say that the state distribution should be as close as possible, and anybody with some RL notion considers DQN as "off policy"  
  
  ---
  
  