Answers.Body: <p>The replay buffer allows breaking the temporal dependence of the data and thus makes them more i.i.d. (which is what we want). The replay buffer needs to be at least filled with enough experience to compute a single batch, but can also be filled completely before the actual training begins (which avoids 'oversampling' the early experiences). Those implementation details were probably left out in most methods sections.</p>
  <p>This question <a href="https://datascience.stackexchange.com/questions/20535/what-is-experience-replay-and-what-are-its-benefits">here</a> provides further details on the replay buffer in the context of DQNs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-29T02:02:46.940Z
Answers.Text: Thank you for the answer. "(which avoids 'oversampling' the early experiences", do you have any source for that?
Answers.UserDisplayName: desert_ranger
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-31T12:25:40.357Z
Answers.Text: I think this follows from the following: Let's say our experiences are in a bag of size N, and currently, we have k (k << N) items in that bag. Each time we sample (e.g. 1 element as batch size) from the bag we also add one new element to the bag (Sampling with Replacement + adding items). After e.g. 1000 iterations we stop. The experiences we added last are the ones we will have sampled the least, the ones, originally in the bag, the most. How much we oversample depends on how different k and N are. Does that make sense?
Answers.UserDisplayName: Mariusmarten
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-28T20:46:44.610Z
Answers.LastActivityDate: 2022-07-28T20:46:44.610Z
Answers.UserDisplayName: Mariusmarten
Body: <p>I have seen many popular RL libraries have a learning_start parameter. This allows the agent to collect enough experiences before training on the replay_buffer. However, I am unable to find the paper that introduced this parameter. This <a href="https://datascience.stackexchange.com/questions/46056/in-keras-library-what-is-the-meaning-of-nb-steps-warmup-in-the-dqnagent-objec">post</a> describes the parameter quite well.</p>
  <p>This <a href="https://arxiv.org/pdf/1706.02677.pdf" rel="nofollow noreferrer">paper</a> does go a little into this parameter but it seems to be in the context of deep learning and not reinforcement learning.</p>
  
Tags: <reinforcement-learning><deep-rl><dqn><hyper-parameters>
Title: Which paper describes the effect of learning_starts in Reinforcement Learning?
UserDisplayName: desert_ranger
ThreadText: # Which paper describes the effect of learning_starts in Reinforcement Learning?
  
  (Asked by: username_0 on 2022-07-26)
  
  I have seen many popular RL libraries have a learning\_start parameter. This allows the agent to collect enough experiences before training on the replay\_buffer. However, I am unable to find the paper that introduced this parameter. This [post](https://datascience.stackexchange.com/questions/46056/in-keras-library-what-is-the-meaning-of-nb-steps-warmup-in-the-dqnagent-objec) describes the parameter quite well.
  
  This [paper](https://arxiv.org/pdf/1706.02677.pdf) does go a little into this parameter but it seems to be in the context of deep learning and not reinforcement learning.
  
  **Tags:** reinforcement-learning, deep-rl, dqn, hyper-parameters
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-28. Score: 2 | Accepted answer
  
  The replay buffer allows breaking the temporal dependence of the data and thus makes them more i.i.d. (which is what we want). The replay buffer needs to be at least filled with enough experience to compute a single batch, but can also be filled completely before the actual training begins (which avoids 'oversampling' the early experiences). Those implementation details were probably left out in most methods sections.
  
  This question [here](https://datascience.stackexchange.com/questions/20535/what-is-experience-replay-and-what-are-its-benefits) provides further details on the replay buffer in the context of DQNs.
  
  > **Comments:**
  > **username_0**: Thank you for the answer. "(which avoids 'oversampling' the early experiences", do you have any source for that?  
  > **username_1**: I think this follows from the following: Let's say our experiences are in a bag of size N, and currently, we have k (k << N) items in that bag. Each time we sample (e.g. 1 element as batch size) from the bag we also add one new element to the bag (Sampling with Replacement + adding items). After e.g. 1000 iterations we stop. The experiences we added last are the ones we will have sampled the least, the ones, originally in the bag, the most. How much we oversample depends on how different k and N are. Does that make sense?  
  
  ---
  
  