Answers.Body: <p>I read the same thing recently, and my interpretation was this:</p>
  <blockquote>
  <p>If you only use the very-most recent data, you will overfit to that and things will break</p>
  </blockquote>
  <p>We'd like to train the network to predict accurately for all states and actions. If we only show it recent data, we're only showing it a subset of the possible states and actions, so it might overfit those and forget how to predict in other parts of the environment.</p>
  <p>Similar to training a model to classify images of cats, dogs, and elephants, but only showing it cats and dogs for a while. It can forget what elephants look like.</p>
  <blockquote>
  <p>if you use too much experience, you may slow down your learning.</p>
  </blockquote>
  <p>Specifically, I believe, when encountering novel data. With more memory, the novel data will be chosen for updates less often. The network will learn more slowly, because it spends much of its time reviewing old data that it has already learned.</p>
  <p>The optimal balance feeds the network as much new data as possible while feeding it enough old data so it doesn't forget.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-16T01:42:26.027Z
Answers.Text: this is helpful in general. But could you respond to the specific points I raised as well?
Answers.UserDisplayName: Sara
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-16T02:35:08.263Z
Answers.Text: I believe the points you made are accurate regarding convergence of a NN to fixed training data, but this training data is not fixed. In later learning steps the NN has already converged on the majority of states (losses are near zero), and the NN is only learning from a few novel states. Keeping old data slows down training because it reduces how often the only states the network can learn from (the novel states) are included in a minibatch. (At least that's my understanding.)
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-15T01:40:18.687Z
Answers.LastActivityDate: 2022-06-15T01:40:18.687Z
Answers.UserDisplayName: Lee Reeves
Body: <p><a href="https://spinningup.openai.com/en/latest/algorithms/ddpg.html#deep-deterministic-policy-gradient" rel="nofollow noreferrer">Open AI spin up</a> says</p>
  <blockquote>
  <p>... the replay buffer should be large enough to contain a wide range
  of experiences, but it may not always be good to keep everything. If
  you only use the very-most recent data, you will overfit to that and
  things will break; if you use too much experience, you may slow down
  your learning.</p>
  </blockquote>
  <ol>
  <li><p>Why would recent data overfit? Is it because they are more correlated to each other so I am training on a bunch of similar data?</p>
  </li>
  <li><p>Why is it slow to use much memory (besides the simple reason that I am storing more data). Below are specific doubts I have.</p>
  <ol>
  <li><p>I thought convergence is faster when fitting on uncorrelated data. For example, estimates on mean. Recent data would be more correlated. So why would training on longer data slow down training?</p>
  </li>
  <li><p>Previous samples doesn't seem to have the notion of becoming outdated. They are still valid samples for evaluating the current loss. So why would they slow down training?</p>
  </li>
  </ol>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-14T10:34:30.407Z
Comments.Text: Please, ask only one question per post. If you have multiple questions, ask each in its separate post with the appropriate context. I see many `?` here, so it seems you're asking many questions. If you have one main question and the other are just sub-questions that help people answer your main question, then you should probably clarify that. I suggest that you also fix the title, which ends with "and", so it seems that you missed something there.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-15T09:14:10.463Z
Comments.Text: Your question(s) is/are related to [this](https://ai.stackexchange.com/q/11640/2444).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-16T01:30:01.707Z
Comments.Text: Thanks for your advice on the post, I edited the question and took out some part for another post. I did read the question you referenced, but although it is on the same topic, the answers there didn't solved the question for me, so I try to list out specific points I am confused about here.
Comments.UserDisplayName: Sara
Tags: <reinforcement-learning><deep-rl><hyper-parameters><experience-replay>
Title: Why is a large replay buffer inefficient?
UserDisplayName: Sara
ThreadText: # Why is a large replay buffer inefficient?
  
  (Asked by: username_0 on 2022-06-14)
  
  [Open AI spin up](https://spinningup.openai.com/en/latest/algorithms/ddpg.html#deep-deterministic-policy-gradient) says
  
  > ... the replay buffer should be large enough to contain a wide range
  > of experiences, but it may not always be good to keep everything. If
  > you only use the very-most recent data, you will overfit to that and
  > things will break; if you use too much experience, you may slow down
  > your learning.
  
  1. Why would recent data overfit? Is it because they are more correlated to each other so I am training on a bunch of similar data?
  2. Why is it slow to use much memory (besides the simple reason that I am storing more data). Below are specific doubts I have.
  
     1. I thought convergence is faster when fitting on uncorrelated data. For example, estimates on mean. Recent data would be more correlated. So why would training on longer data slow down training?
     2. Previous samples doesn't seem to have the notion of becoming outdated. They are still valid samples for evaluating the current loss. So why would they slow down training?
  
  **Tags:** reinforcement-learning, deep-rl, hyper-parameters, experience-replay
  > **Comments:**
  > **username_1**: Please, ask only one question per post. If you have multiple questions, ask each in its separate post with the appropriate context. I see many `?` here, so it seems you're asking many questions. If you have one main question and the other are just sub-questions that help people answer your main question, then you should probably clarify that. I suggest that you also fix the title, which ends with "and", so it seems that you missed something there.  
  > **username_1**: Your question(s) is/are related to [this](https://ai.stackexchange.com/q/11640/2444).  
  > **username_0**: Thanks for your advice on the post, I edited the question and took out some part for another post. I did read the question you referenced, but although it is on the same topic, the answers there didn't solved the question for me, so I try to list out specific points I am confused about here.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-06-15. Score: 3
  
  I read the same thing recently, and my interpretation was this:
  
  > If you only use the very-most recent data, you will overfit to that and things will break
  
  We'd like to train the network to predict accurately for all states and actions. If we only show it recent data, we're only showing it a subset of the possible states and actions, so it might overfit those and forget how to predict in other parts of the environment.
  
  Similar to training a model to classify images of cats, dogs, and elephants, but only showing it cats and dogs for a while. It can forget what elephants look like.
  
  > if you use too much experience, you may slow down your learning.
  
  Specifically, I believe, when encountering novel data. With more memory, the novel data will be chosen for updates less often. The network will learn more slowly, because it spends much of its time reviewing old data that it has already learned.
  
  The optimal balance feeds the network as much new data as possible while feeding it enough old data so it doesn't forget.
  
  > **Comments:**
  > **username_0**: this is helpful in general. But could you respond to the specific points I raised as well?  
  > **username_2**: I believe the points you made are accurate regarding convergence of a NN to fixed training data, but this training data is not fixed. In later learning steps the NN has already converged on the majority of states (losses are near zero), and the NN is only learning from a few novel states. Keeping old data slows down training because it reduces how often the only states the network can learn from (the novel states) are included in a minibatch. (At least that's my understanding.)  
  
  ---
  
  