Answers.Body: <p>You need to read this 2020 paper by Deepmind:
  <a href="http://acsweb.ucsd.edu/%7Ewfedus/pdf/replay.pdf" rel="nofollow noreferrer">&quot;Revisiting Fundamentals of Experience Replay&quot;</a>
  They explicitly test the size of the experience replay, the replay-ratio of each experience and other parameters.</p>
  <p>Also, to add to the answer by @nbro</p>
  <p>Assume you implement experience replay as a buffer where the newest memory is stored instead of the oldest. Then, if your buffer contains 100k entries, any memory will remain there for exactly 100k iterations.</p>
  <p>Such a buffer is simply a way to &quot;see&quot; what was up to 100k iterations ago.
  After the first 100k iterations you fill the buffer and begin &quot;moving&quot; it, much like a sliding window, by inserting new memories instead of the oldest.</p>
  <hr />
  <p>The size of the buffer (relative to the total number of iterations you plan to ever train with) depends on &quot;how much you believe your network architecture is susceptible to <a href="https://ai.stackexchange.com/a/13293/27042">catastrophic forgetting</a>&quot;.</p>
  <p>A tiny buffer might force your network to only care about what it saw recently.</p>
  <p>But an excessively large buffer might take a long time to &quot;become refreshed&quot; with good trajectories, when they finally start to be discovered. So the network would be like a university student whose book shelf is diluted with first-grade school books.</p>
  <p>The student might have already decided that he/she wishes to become a programmer, so re-reading those primary school books has little benefit (time could have been spent more productively on programming literature) + it takes a long time to replace those with relevant university books.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-10T10:46:29.793Z
Answers.Text: Good point. Just to add to why this is important: this is important because old experience was generated by an old policy (i.e. your data distribution consists of an older distribution over state-action pairs / trajectories), and because any stored value estimates may be old. Which one of those two points matters depends on whether we're talking about value-based or policy gradient methods, and whether we're talking about on-policy or off-policy learning... but usually at least one of those two is relevant
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-10T11:39:20.833Z
Answers.Text: @DennisSoemers: Generally you should not store value estimates in experience replay, but re-calculate values when using the trajectory information (s,a,r,s'). No doubt there are implementations which store value estimates for some valid reason, but "classic" DQN does not
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-10T11:51:43.113Z
Answers.Text: @NeilSlater Yeah you're right, I guess you shouldn't in most (all?) of the classic RL algorithms. I've been working a lot with AlphaZero-style self-playing training the past year, and there we do store value estimates as computed by MCTS... so I guess that's why I've got it stuck in my mind :)
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-10T11:57:41.643Z
Answers.Text: No wait that's wrong too, in standard AlphaZero-style training we store visit counts, but in one of my recent papers I changed that to storing MCTS' value estimates...
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-10T10:41:02.730Z
Answers.LastActivityDate: 2023-05-02T17:36:23.360Z
Answers.LastEditDate: 2023-05-02T17:36:23.360Z
Answers.UserDisplayName: Kari
Answers.Body: <blockquote>
    <p>In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything.</p>
  </blockquote>
  
  <p>The larger the experience replay, the less likely you will sample correlated elements, hence the more stable the training of the NN will be. However, a large experience replay also requires a lot of memory and it might slow training. So, there is a trade-off between training stability (of the NN) and memory requirements.</p>
  
  <p>The authors of the linked article state (right after the sentence above)</p>
  
  <blockquote>
    <p>If you only use the very-most recent data, you will overfit to that and things will break; if you use too much experience, you may slow down your learning. This may take some tuning to get right.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-04T16:24:25.587Z
Answers.LastActivityDate: 2019-07-10T17:45:45.407Z
Answers.LastEditDate: 2019-07-10T17:45:45.407Z
Answers.UserDisplayName: nbro
Body: <p>I'm learning DDPG algorithm by following the following link: <a href="https://spinningup.openai.com/en/latest/algorithms/ddpg.html#the-policy-learning-side-of-ddpg" rel="noreferrer">Open AI Spinning Up document on DDPG</a>, where it is written</p>
  
  <blockquote>
    <p>In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. </p>
  </blockquote>
  
  <p>What does this mean? Is it related to the tuning of the parameter of the batch size in the algorithm?</p>
  
Tags: <reinforcement-learning><deep-rl><hyper-parameters><ddpg><experience-replay>
Title: How large should the replay buffer be?
UserDisplayName: ycenycute
ThreadText: # How large should the replay buffer be?
  
  (Asked by: username_0 on 2019-04-04)
  
  I'm learning DDPG algorithm by following the following link: [Open AI Spinning Up document on DDPG](https://spinningup.openai.com/en/latest/algorithms/ddpg.html#the-policy-learning-side-of-ddpg), where it is written
  
  > In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything.
  
  What does this mean? Is it related to the tuning of the parameter of the batch size in the algorithm?
  
  **Tags:** reinforcement-learning, deep-rl, hyper-parameters, ddpg, experience-replay
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-07-10. Score: 11
  
  You need to read this 2020 paper by Deepmind:
  ["Revisiting Fundamentals of Experience Replay"](http://acsweb.ucsd.edu/%7Ewfedus/pdf/replay.pdf)
  They explicitly test the size of the experience replay, the replay-ratio of each experience and other parameters.
  
  Also, to add to the answer by @username_4
  
  Assume you implement experience replay as a buffer where the newest memory is stored instead of the oldest. Then, if your buffer contains 100k entries, any memory will remain there for exactly 100k iterations.
  
  Such a buffer is simply a way to "see" what was up to 100k iterations ago.
  After the first 100k iterations you fill the buffer and begin "moving" it, much like a sliding window, by inserting new memories instead of the oldest.
  
  ---
  
  The size of the buffer (relative to the total number of iterations you plan to ever train with) depends on "how much you believe your network architecture is susceptible to [catastrophic forgetting](https://ai.stackexchange.com/a/13293/27042)".
  
  A tiny buffer might force your network to only care about what it saw recently.
  
  But an excessively large buffer might take a long time to "become refreshed" with good trajectories, when they finally start to be discovered. So the network would be like a university student whose book shelf is diluted with first-grade school books.
  
  The student might have already decided that he/she wishes to become a programmer, so re-reading those primary school books has little benefit (time could have been spent more productively on programming literature) + it takes a long time to replace those with relevant university books.
  
  > **Comments:**
  > **username_2**: Good point. Just to add to why this is important: this is important because old experience was generated by an old policy (i.e. your data distribution consists of an older distribution over state-action pairs / trajectories), and because any stored value estimates may be old. Which one of those two points matters depends on whether we're talking about value-based or policy gradient methods, and whether we're talking about on-policy or off-policy learning... but usually at least one of those two is relevant  
  > **username_3**: @username_2: Generally you should not store value estimates in experience replay, but re-calculate values when using the trajectory information (s,a,r,s'). No doubt there are implementations which store value estimates for some valid reason, but "classic" DQN does not  
  > **username_2**: @username_3 Yeah you're right, I guess you shouldn't in most (all?) of the classic RL algorithms. I've been working a lot with AlphaZero-style self-playing training the past year, and there we do store value estimates as computed by MCTS... so I guess that's why I've got it stuck in my mind :)  
  > **username_2**: No wait that's wrong too, in standard AlphaZero-style training we store visit counts, but in one of my recent papers I changed that to storing MCTS' value estimates...  
  
  ---
  
  ### Answer by username_4 on 2019-04-04. Score: 10
  
  > In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything.
  
  The larger the experience replay, the less likely you will sample correlated elements, hence the more stable the training of the NN will be. However, a large experience replay also requires a lot of memory and it might slow training. So, there is a trade-off between training stability (of the NN) and memory requirements.
  
  The authors of the linked article state (right after the sentence above)
  
  > If you only use the very-most recent data, you will overfit to that and things will break; if you use too much experience, you may slow down your learning. This may take some tuning to get right.
  
  
  ---
  
  