Answers.Body: <blockquote>
  <p>The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right?</p>
  </blockquote>
  <p>Yes that is a major benefit of using a replay buffer.</p>
  <p>A secondary benefit is the ability to use the same sample more than once. This can lead to beter sample efficiency, although that is not guaranteed.</p>
  <blockquote>
  <p>What would happen if we calculate our loss using just one experience instead of a mini-batch of experiences?</p>
  </blockquote>
  <p>The algorithm is still valid, but the gradient estimate for the update step would be based on a single record of [state, action, reward, next state]. This would be a high variance update process, with many steps in wrong directions, but in expectation over many steps you should still see a correct gradient. You would probably need to compensate for the high variance per sample by reducing the learning rate.</p>
  <p>In addition, assuming the standard approach of collecting one time step then making one update to DQN neural network, each piece of experience would only be used once on average before being discarded.</p>
  <p>These two effects will likely combine such that the learning process would not be very sample efficient.</p>
  <p>The size of the minibatch is one of many hyperparameters you can change in DQN. It might be the case for some problems that choosing a low minibatch size is helpful, provided other adjustments (such as a lower learning rate) are made along with it. If you are not sure, you mostly have to try and see.</p>
  <p>In my experience on a small range of problems, a moderate size of minibatch - ranging from 10 to 100 - has worked the best in terms of end results of high scoring agents. However, I have not spent a long time trying to make low batch sizes work.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-22T20:40:16.477Z
Answers.Text: Can you simplify what you mean by a high variance update ?, does this mean it would give a higher divergence than when a batch is used?
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-22T21:01:42.903Z
Answers.Text: @Chukwudi: High variance = "varies a lot". Statistcally in the sense of mean and variance. If you take more samples and take the mean (which is what a batch does), then the variance of the expected mean is divided by the number of samples.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-22T21:03:06.800Z
Answers.Text: Oh so the higher number of samples the more accurate the measurements, so the lower the variance right
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-22T21:05:11.540Z
Answers.Text: @Chukwudi: Yes. But of course if takes more time and effort to collect those samples. Meaning you take longer to make updates to your NN. So it is not always clear whether to take small or large batches.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-22T20:09:05.673Z
Answers.LastActivityDate: 2020-06-22T20:09:05.673Z
Answers.UserDisplayName: Neil Slater
Body: <p>The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right?</p>
  <p>What would happen if we calculate our loss using just one experience instead of a mini-batch of experiences?</p>
  
Tags: <reinforcement-learning><q-learning><deep-rl><experience-replay>
Title: What would happen if we sampled only one tuple from the experience replay?
UserDisplayName: Chukwudi
ThreadText: # What would happen if we sampled only one tuple from the experience replay?
  
  (Asked by: username_0 on 2020-06-22)
  
  The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right?
  
  What would happen if we calculate our loss using just one experience instead of a mini-batch of experiences?
  
  **Tags:** reinforcement-learning, q-learning, deep-rl, experience-replay
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-22. Score: 0 | Accepted answer
  
  > The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right?
  
  Yes that is a major benefit of using a replay buffer.
  
  A secondary benefit is the ability to use the same sample more than once. This can lead to beter sample efficiency, although that is not guaranteed.
  
  > What would happen if we calculate our loss using just one experience instead of a mini-batch of experiences?
  
  The algorithm is still valid, but the gradient estimate for the update step would be based on a single record of [state, action, reward, next state]. This would be a high variance update process, with many steps in wrong directions, but in expectation over many steps you should still see a correct gradient. You would probably need to compensate for the high variance per sample by reducing the learning rate.
  
  In addition, assuming the standard approach of collecting one time step then making one update to DQN neural network, each piece of experience would only be used once on average before being discarded.
  
  These two effects will likely combine such that the learning process would not be very sample efficient.
  
  The size of the minibatch is one of many hyperparameters you can change in DQN. It might be the case for some problems that choosing a low minibatch size is helpful, provided other adjustments (such as a lower learning rate) are made along with it. If you are not sure, you mostly have to try and see.
  
  In my experience on a small range of problems, a moderate size of minibatch - ranging from 10 to 100 - has worked the best in terms of end results of high scoring agents. However, I have not spent a long time trying to make low batch sizes work.
  
  > **Comments:**
  > **username_0**: Can you simplify what you mean by a high variance update ?, does this mean it would give a higher divergence than when a batch is used?  
  > **username_1**: @username_0: High variance = "varies a lot". Statistcally in the sense of mean and variance. If you take more samples and take the mean (which is what a batch does), then the variance of the expected mean is divided by the number of samples.  
  > **username_0**: Oh so the higher number of samples the more accurate the measurements, so the lower the variance right  
  > **username_1**: @username_0: Yes. But of course if takes more time and effort to collect those samples. Meaning you take longer to make updates to your NN. So it is not always clear whether to take small or large batches.  
  
  ---
  
  