Answers.Body: <p>This question is asking about randomness in the data, so I will first clarify one of the comments in this question made about randomness.</p>
  <hr />
  <blockquote>
  <p>If the data is taken randomly, how can a DQN approximate a meaningful function? It would be like going into a labyrinth marking the way and the directions chosen along the way. Then, after a while (an episode in our programming case), all those markings are randomly changing their direction/position.</p>
  </blockquote>
  <p>This phenomenon is a fundamental consideration in the design of reinforcement learning algorithms as opposed to other learning algorithms. For example, supervised learning algorithms would learn to map states to actions solely through the state and corresponding action pairs provided in a fixed dataset. Like you mentioned, any randomness in the corresponding action will generally degrade the learning process.</p>
  <p>In contrast, reinforcement learning algorithms do not not assume to have any knowledge of the desired actions beforehand and <strong>must learn the desired actions through interacting with the environment</strong> and receiving a reward signal and a next state signal. Consequently, <strong>randomness or variation in the actions is necessary to investigate the possible actions</strong> to see which actions will accrue the most reward; training on the data will increase the probability of taking actions that accrued more reward, and vice versa. In other words, the training process of reinforcement learning is meant to assess the data instead of imitate a fixed dataset as in supervised learning.</p>
  <hr />
  <blockquote>
  <p>All those transitions are stored in one Python deque (or a long memory buffer). And now the surprise to me: one a small amount of those data taken from that buffer. Much worse, <em>randomly</em>. For my limited intuition capacity, this is hard to believe. Because due to the randomness of this batch, the sequence of actions played in one episode doesn't play any role anymore. And this is my question: How it is supposed to work?</p>
  </blockquote>
  <p>To understand <em>why training on sequences of actions is unnecessary</em>, it helps to understand <em>why it is possible to train on old data</em> using the Q-learning update (see equation 6.8 of <a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">Sutton and Barto's book</a>):</p>
  <p><span class="math-container">$$Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha\left[(R_{t+1} + \gamma \max_{a}Q(S_{t+1}, a)) - Q(S_t, A_t)\right].$$</span></p>
  <p>Here, the approximate Q-function <span class="math-container">$Q(S_t, A_t)$</span> is being updated with the state, action, reward, next state tuples <span class="math-container">$(s, a, r, s')$</span>. Importantly, these tuples were generated by a policy that takes random actions a small percentage of the time (<span class="math-container">$\epsilon$</span>-greedy exploration). This policy that generates the data is the <em>behavior policy</em>. In contrast, the policy being learned is the policy that never explores (and only exploits) through using <span class="math-container">$\underset{a}{\max}Q(S_{t+1}, a)$</span> as an estimate of the return from the next state. This policy that only exploits is called the <em>target policy</em> that doesn't match the behavior policy of taking exploratory actions. Under certain conditions that I won't detail (e.g. reducing the exploration hyperparameter <span class="math-container">$\epsilon$</span>), successively performing Q-learning updates is shown to converge to the optimal Q-function. Therefore, old data from previous behavior policies can be used during the Q-learning update to approximate the target policy.</p>
  <p>The above paragraph should answer the question of <em>why old data can be used</em> in the buffer. Now we address the question of <em>why the data can be sampled randomly</em> without regard to its original sequence. In the Q-learning update presented above, only a single <span class="math-container">$(s, a, r, s')$</span> experience tuple is used to update a single <span class="math-container">$(s, a)$</span> pair of the Q-function. The Q-learning update does not need to process an entire sequence of actions in order because <strong>it only looks one time step into the future and approximates the return from all future time steps using the current Q-function</strong>, making it a one-step method or one-step update rule. This provides a great benefit over learning from sequences, as the Q-learning update can <strong>assess the best action at each individual time step</strong>. This is in contrast to training only on good episodes, since <strong>it provides much more flexibility when assigning credit to actions</strong>. For example, algorithms need to take exploratory actions in the beginning of training to find the most return. Regardless of how the remainder of the episode turns out, Q-learning can assess the goodness of a single experience tuple that took an exploratory action by determining if the sum of immediate reward and the expected return from the next state is better than the expected return from current state (via the update rule above). If so, Q-function will learn to take that action more frequently, and vice versa. In summary, Q-learning is able to assess the contribution of individual actions on a time step scale of length 1 through approximating the return from the next state, preventing it from needing to train on sequences of data.</p>
  <p>The final question that remains is <em>if there is any benefit to training with random data</em> as opposed to sequentially experienced data. Taking a look at the <a href="https://arxiv.org/pdf/1312.5602.pdf" rel="nofollow noreferrer">DQN paper</a> (see section 4, paragraph 4), the primary reason to randomly sample past data is to break correlations in the data. First, the high variance of strongly correlated experience hinders learning when using neural networks and associated optimizers. Randomizing the data makes it much closer to i.i.d. data, which generally enables much more efficient training. Second, note that the data used to train the current policy is also generated by the current policy when not using an experience replay buffer. Averaging the learning by randomly sampling data from many previous policies by using an experience replay buffer can help prevent major oscillations or even complete divergence in the learned policy (refer to the paper for a more detailed discussion).</p>
  <p>If any of the terms used in this answer do not make sense and you cannot find a suitable reference online, please post another comment and I can update this answer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-16T07:31:01.177Z
Answers.Text: I think the Bellman equation is key here to OP's understanding - it adds the fact that you *can* learn a useful function using only a small part of the trajectory. Add in the need for iid data for a neural network, and that *can* becomes a *should* .
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-16T14:34:17.427Z
Answers.Text: @NeilSlater You're absolutely right that some additional comments regarding the Bellman equation would most likely increase the quality of my answer. It was somewhat hard to answer this question since (i) the question posed in the title seems to be related but slightly different from the question in the main body and (ii) OP wanted a figurative explanation. My goal with this answer was to help OP understand how RL assesses data instead of imitates it. Do you think my answer adds sufficient value, or would you recommend that I include additional information on the Bellman equation?
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-16T19:21:53.307Z
Answers.Text: My suggestion is based on the OP wanting to know why sampling randomly from the "memory buffer" (replay memory) is effective. This seems the strongest theme in their question. To me, the most salient points are as per my first comment. Namely that the Bellman equation links data from $t$ and $t+1$ time steps and needs no other data from any trajectory, and that neural networks *prefer* iid data which is not correlated (as it would be if you trained strictly in sequence)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-16T19:45:37.103Z
Answers.Text: @NeilSlater I see. I tried to answer more of the first question (regarding the function approximation and training on random data) since it is very important in understanding the second question. I'm starting to realize more and more why we try to limit posts to one question. 
  
  Also during my time on this board, I've seen that you have some of the best etiquette and do not 'snipe' answers from other people and instead offer comments for slight improvements. Just to be sure, by giving me this suggestion, do you want me to add on to my answer, or would you like to make a separate answer?
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-16T20:16:36.183Z
Answers.Text: Generally, if there is already a good answer that I mostly agree with, then I prefer to offer a few things that IMO could help, instead of writing my own. It's still just my opinion, and it's your answer. If you don't think it improves things on balance, e.g. it over-complicates the answer, then there's no pressure.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-16T20:51:16.073Z
Answers.Text: Ok guys, thanks for the answers.
  If I understood right: I interpret @DeepQZero 's answer more as "randomness increase robustness of the network".  While NeilSlater seems to have understood my question which is, I put it easly, "why is the batch random?". The problem is now, that I didn't understand NeilSlater's answer. Is this the (or part of) the answer? https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-16T20:57:54.503Z
Answers.Text: @Dave  I'll try to incorporate the Bellman equation into my answer and make edits over the next day. In the meantime, if you would reduce the post to contain a single question, it would help me out.
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-17T14:09:47.120Z
Answers.Text: @DeepQZero I update the original question with more information. And asking only one question
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-17T21:42:26.947Z
Answers.Text: @Dave I updated my answer. If it isn't satisfactory, please let me know which parts could use more explanation. This answer needs to be high-level, and I can point out other written sources if there are still concerns.
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-17T21:43:51.243Z
Answers.Text: @NeilSlater See above. I didn't discuss the Bellman equation, but I think my answer is sufficient. That is about as much detail as I can write as an answer on this site without needing the question to be more focused.
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-18T06:04:27.940Z
Answers.Text: It's all solidly correct as far as I can see, and I don't have any further suggestions.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-16T06:47:06.643Z
Answers.LastActivityDate: 2023-10-17T21:41:11.820Z
Answers.LastEditDate: 2023-10-17T21:41:11.820Z
Answers.UserDisplayName: DeepQZero
Body: <p>I'm trying to understand DQNs.
  There is one concept that I cannot really understand yet. In the book &quot;Introduction to Reinforcement Learning&quot; as well <a href="https://pythonprogramming.net/training-deep-q-learning-dqn-reinforcement-learning-python-tutorial/" rel="nofollow noreferrer">this tutorial</a> online introduce the concept of a memory buffer (most of the time a Python deque) which must be filled with the following data:</p>
  <p><span class="math-container">$S_{t}, a_{t}, r_{t}, S_{t+1}$</span></p>
  <p>where <span class="math-container">$S_{t}$</span> is the current state, <span class="math-container">$r_{t}$</span> is the reward, <span class="math-container">$S_{t+1}$</span> is the next status of the system given the action <span class="math-container">$a_{t}$</span>.</p>
  <p>Now, the buffer will be filled up to a point. Then a <em>batch</em> of data will be taken <em>randomly</em> from the buffer.
  A code representation can be seen here: <a href="https://github.com/DeepReinforcementLearning/DeepReinforcementLearningInAction/blob/master/Chapter%203/Ch3_book.ipynb" rel="nofollow noreferrer">link</a>.</p>
  <pre><code>...
  replay.append(exp) #H
  state1 = state2
          
  if len(replay) &gt; batch_size:
      minibatch = random.sample(replay, batch_size)
      state1_batch = torch.cat([s1 for (s1,a,r,s2,d) in minibatch])
      action_batch = torch.Tensor([a for (s1,a,r,s2,d) in minibatch])
      reward_batch = torch.Tensor([r for (s1,a,r,s2,d) in minibatch])
      state2_batch = torch.cat([s2 for (s1,a,r,s2,d) in minibatch])
      done_batch = torch.Tensor([d for (s1,a,r,s2,d) in minibatch])
      Q1 = model(state1_batch)
  ...
  </code></pre>
  <p>The fact, that a batch is taken randomly is not intuitive for me for understanding the logic behind it. By taking the data randomly it would be very hard to figure out any possible pattern. It would be like going into a labyrinth marking the way and the directions choosen along the way. Then, after a while (an episode in our programming case), all those markings are <em>randomly</em> changing their direction/position.</p>
  <p><s>Maybe I'm wrong with my example, but if the data is taken randomly, how can a DQN approximate a meaningful function?</s></p>
  <p>UPDATE 17 October 2023</p>
  <p>I really thanks @NeilSlater and @DeepQZero for their answers.
  But it seems, that probably my question was not really clear and/or lead to misinterpretations.</p>
  <p>So I would present now an example.
  In many tutorials about DQN, the gymnasium (or gym) environment is going to be used. CartPole, FrozenLake and many more are simple application, where you can develop your DQN network and test it.</p>
  <p>Let's take for instance the <em>CartPole</em> environment. In it you get a +1 as a reward, for every scene, in which the pole remains up within the +/- 12° range.
  So we can imagine to run one episode. In this episode the pole soon or later will definetly falls down leading to the end of the episode.
  In my tests, if my action is completely random, than an episode lasts about 20 steps, then it is over.
  But... it could be, that one episode is particular lucly and that pole will stay vertical for...let's say 50-60 steps (it is just an example).
  In other words, <em>the sequence of actions led to a very high reward</em> (+1 * number of steps). Which is good, because luckly or not, I got a sequence, which can be used to &quot;solve&quot; the next game.</p>
  <p>But now, the problem. All those <em>transistions</em> are stored in one Python deque (or a long memory buffer). And now the surprise to me: <em>one a small amount of those data taken from that buffer</em>. Much worse, <strong>randomly</strong>.
  For my limited intuition capacity, this is hard to believe. Because due to the <strong>randomness</strong> of this <em>batch</em>, the <em>sequence of actions</em> played in one episode doesn't play any role anymore. And this is my question: How it is supposed to work?</p>
  <p>Maybe I'm missing some important points here. But my thought were stronger after I read the book &quot;Deep Reinforcement Learning&quot; by Maxim Lapan. In the very first example, he does the following things:</p>
  <ol>
  <li>run one episode of CartPole</li>
  <li>collect action, state, new state and reward for every step</li>
  <li>create a new deque for every new episode</li>
  <li><strong>take a batch of data containing only the most rewarded episodes</strong> (he used a percentile of 70%)</li>
  <li>train the network</li>
  <li>repeat</li>
  </ol>
  <p>As you can see, he takes a batch based on the most successful episodes.
  That means, the sequence of actions is important.
  But for DQN, the batch is taken randomly as the sequence is no important anymore.</p>
  <p>I hope, I explained my thoughts...</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-15T23:04:14.537Z
Comments.Text: first understand Q-learning, then pass to Deep Q Learning
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-16T20:43:15.753Z
Comments.Text: I do not think, that that is the problem
Comments.UserDisplayName: Dave
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-16T21:39:47.253Z
Comments.Text: "_that a batch is taken randomly is not intuitive for me for understanding the logic behind it_" if you would have studied Q-learning before DQN, you should get why this is fine
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-17T19:40:19.683Z
Comments.Text: For anyone answering, the training method presented for CartPole is correctly described - it's the Cross Entropy Method, which is a form of policy search which is not commonly taught in RL courses. It is its own thing, but is somewhere in-between genetic algorithms and policy gradients conceptually.
Comments.UserDisplayName: Neil Slater
Tags: <dqn><batch-size>
Title: What is the purpose of a replay/memory buffer in Deep Q-Learning networks?
UserDisplayName: Dave
ThreadText: # What is the purpose of a replay/memory buffer in Deep Q-Learning networks?
  
  (Asked by: username_0 on 2023-10-15)
  
  I'm trying to understand DQNs.
  There is one concept that I cannot really understand yet. In the book "Introduction to Reinforcement Learning" as well [this tutorial](https://pythonprogramming.net/training-deep-q-learning-dqn-reinforcement-learning-python-tutorial/) online introduce the concept of a memory buffer (most of the time a Python deque) which must be filled with the following data:
  
  $S\_{t}, a\_{t}, r\_{t}, S\_{t+1}$
  
  where $S\_{t}$ is the current state, $r\_{t}$ is the reward, $S\_{t+1}$ is the next status of the system given the action $a\_{t}$.
  
  Now, the buffer will be filled up to a point. Then a *batch* of data will be taken *randomly* from the buffer.
  A code representation can be seen here: [link](https://github.com/DeepReinforcementLearning/DeepReinforcementLearningInAction/blob/master/Chapter%203/Ch3_book.ipynb).
  
  ```
  ...
  replay.append(exp) #H
  state1 = state2
          
  if len(replay) > batch_size:
      minibatch = random.sample(replay, batch_size)
      state1_batch = torch.cat([s1 for (s1,a,r,s2,d) in minibatch])
      action_batch = torch.Tensor([a for (s1,a,r,s2,d) in minibatch])
      reward_batch = torch.Tensor([r for (s1,a,r,s2,d) in minibatch])
      state2_batch = torch.cat([s2 for (s1,a,r,s2,d) in minibatch])
      done_batch = torch.Tensor([d for (s1,a,r,s2,d) in minibatch])
      Q1 = model(state1_batch)
  ...
  
  ```
  
  The fact, that a batch is taken randomly is not intuitive for me for understanding the logic behind it. By taking the data randomly it would be very hard to figure out any possible pattern. It would be like going into a labyrinth marking the way and the directions choosen along the way. Then, after a while (an episode in our programming case), all those markings are *randomly* changing their direction/position.
  
  ~~Maybe I'm wrong with my example, but if the data is taken randomly, how can a DQN approximate a meaningful function?~~
  
  UPDATE 17 October 2023
  
  I really thanks @username_2 and @username_3 for their answers.
  But it seems, that probably my question was not really clear and/or lead to misinterpretations.
  
  So I would present now an example.
  In many tutorials about DQN, the gymnasium (or gym) environment is going to be used. CartPole, FrozenLake and many more are simple application, where you can develop your DQN network and test it.
  
  Let's take for instance the *CartPole* environment. In it you get a +1 as a reward, for every scene, in which the pole remains up within the +/- 12° range.
  So we can imagine to run one episode. In this episode the pole soon or later will definetly falls down leading to the end of the episode.
  In my tests, if my action is completely random, than an episode lasts about 20 steps, then it is over.
  But... it could be, that one episode is particular lucly and that pole will stay vertical for...let's say 50-60 steps (it is just an example).
  In other words, *the sequence of actions led to a very high reward* (+1 \* number of steps). Which is good, because luckly or not, I got a sequence, which can be used to "solve" the next game.
  
  But now, the problem. All those *transistions* are stored in one Python deque (or a long memory buffer). And now the surprise to me: *one a small amount of those data taken from that buffer*. Much worse, **randomly**.
  For my limited intuition capacity, this is hard to believe. Because due to the **randomness** of this *batch*, the *sequence of actions* played in one episode doesn't play any role anymore. And this is my question: How it is supposed to work?
  
  Maybe I'm missing some important points here. But my thought were stronger after I read the book "Deep Reinforcement Learning" by Maxim Lapan. In the very first example, he does the following things:
  
  1. run one episode of CartPole
  2. collect action, state, new state and reward for every step
  3. create a new deque for every new episode
  4. **take a batch of data containing only the most rewarded episodes** (he used a percentile of 70%)
  5. train the network
  6. repeat
  
  As you can see, he takes a batch based on the most successful episodes.
  That means, the sequence of actions is important.
  But for DQN, the batch is taken randomly as the sequence is no important anymore.
  
  I hope, I explained my thoughts...
  
  **Tags:** dqn, batch-size
  
  > **Comments:**
  > **username_1**: first understand Q-learning, then pass to Deep Q Learning  
  > **username_0**: I do not think, that that is the problem  
  > **username_1**: "_that a batch is taken randomly is not intuitive for me for understanding the logic behind it_" if you would have studied Q-learning before DQN, you should get why this is fine  
  > **username_2**: For anyone answering, the training method presented for CartPole is correctly described - it's the Cross Entropy Method, which is a form of policy search which is not commonly taught in RL courses. It is its own thing, but is somewhere in-between genetic algorithms and policy gradients conceptually.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-10-16. Score: 4 | Accepted answer
  
  This question is asking about randomness in the data, so I will first clarify one of the comments in this question made about randomness.
  
  ---
  
  > If the data is taken randomly, how can a DQN approximate a meaningful function? It would be like going into a labyrinth marking the way and the directions chosen along the way. Then, after a while (an episode in our programming case), all those markings are randomly changing their direction/position.
  
  This phenomenon is a fundamental consideration in the design of reinforcement learning algorithms as opposed to other learning algorithms. For example, supervised learning algorithms would learn to map states to actions solely through the state and corresponding action pairs provided in a fixed dataset. Like you mentioned, any randomness in the corresponding action will generally degrade the learning process.
  
  In contrast, reinforcement learning algorithms do not not assume to have any knowledge of the desired actions beforehand and **must learn the desired actions through interacting with the environment** and receiving a reward signal and a next state signal. Consequently, **randomness or variation in the actions is necessary to investigate the possible actions** to see which actions will accrue the most reward; training on the data will increase the probability of taking actions that accrued more reward, and vice versa. In other words, the training process of reinforcement learning is meant to assess the data instead of imitate a fixed dataset as in supervised learning.
  
  ---
  
  > All those transitions are stored in one Python deque (or a long memory buffer). And now the surprise to me: one a small amount of those data taken from that buffer. Much worse, *randomly*. For my limited intuition capacity, this is hard to believe. Because due to the randomness of this batch, the sequence of actions played in one episode doesn't play any role anymore. And this is my question: How it is supposed to work?
  
  To understand *why training on sequences of actions is unnecessary*, it helps to understand *why it is possible to train on old data* using the Q-learning update (see equation 6.8 of [Sutton and Barto's book](http://incompleteideas.net/book/the-book-2nd.html)):
  
  $$Q(S\_t, A\_t) \leftarrow Q(S\_t, A\_t) + \alpha\left[(R\_{t+1} + \gamma \max\_{a}Q(S\_{t+1}, a)) - Q(S\_t, A\_t)\right].$$
  
  Here, the approximate Q-function $Q(S\_t, A\_t)$ is being updated with the state, action, reward, next state tuples $(s, a, r, s')$. Importantly, these tuples were generated by a policy that takes random actions a small percentage of the time ($\epsilon$-greedy exploration). This policy that generates the data is the *behavior policy*. In contrast, the policy being learned is the policy that never explores (and only exploits) through using $\underset{a}{\max}Q(S\_{t+1}, a)$ as an estimate of the return from the next state. This policy that only exploits is called the *target policy* that doesn't match the behavior policy of taking exploratory actions. Under certain conditions that I won't detail (e.g. reducing the exploration hyperparameter $\epsilon$), successively performing Q-learning updates is shown to converge to the optimal Q-function. Therefore, old data from previous behavior policies can be used during the Q-learning update to approximate the target policy.
  
  The above paragraph should answer the question of *why old data can be used* in the buffer. Now we address the question of *why the data can be sampled randomly* without regard to its original sequence. In the Q-learning update presented above, only a single $(s, a, r, s')$ experience tuple is used to update a single $(s, a)$ pair of the Q-function. The Q-learning update does not need to process an entire sequence of actions in order because **it only looks one time step into the future and approximates the return from all future time steps using the current Q-function**, making it a one-step method or one-step update rule. This provides a great benefit over learning from sequences, as the Q-learning update can **assess the best action at each individual time step**. This is in contrast to training only on good episodes, since **it provides much more flexibility when assigning credit to actions**. For example, algorithms need to take exploratory actions in the beginning of training to find the most return. Regardless of how the remainder of the episode turns out, Q-learning can assess the goodness of a single experience tuple that took an exploratory action by determining if the sum of immediate reward and the expected return from the next state is better than the expected return from current state (via the update rule above). If so, Q-function will learn to take that action more frequently, and vice versa. In summary, Q-learning is able to assess the contribution of individual actions on a time step scale of length 1 through approximating the return from the next state, preventing it from needing to train on sequences of data.
  
  The final question that remains is *if there is any benefit to training with random data* as opposed to sequentially experienced data. Taking a look at the [DQN paper](https://arxiv.org/pdf/1312.5602.pdf) (see section 4, paragraph 4), the primary reason to randomly sample past data is to break correlations in the data. First, the high variance of strongly correlated experience hinders learning when using neural networks and associated optimizers. Randomizing the data makes it much closer to i.i.d. data, which generally enables much more efficient training. Second, note that the data used to train the current policy is also generated by the current policy when not using an experience replay buffer. Averaging the learning by randomly sampling data from many previous policies by using an experience replay buffer can help prevent major oscillations or even complete divergence in the learned policy (refer to the paper for a more detailed discussion).
  
  If any of the terms used in this answer do not make sense and you cannot find a suitable reference online, please post another comment and I can update this answer.
  
  > **Comments:**
  > **username_2**: I think the Bellman equation is key here to OP's understanding - it adds the fact that you *can* learn a useful function using only a small part of the trajectory. Add in the need for iid data for a neural network, and that *can* becomes a *should* .  
  > **username_3**: @username_2 You're absolutely right that some additional comments regarding the Bellman equation would most likely increase the quality of my answer. It was somewhat hard to answer this question since (i) the question posed in the title seems to be related but slightly different from the question in the main body and (ii) OP wanted a figurative explanation. My goal with this answer was to help OP understand how RL assesses data instead of imitates it. Do you think my answer adds sufficient value, or would you recommend that I include additional information on the Bellman equation?  
  > **username_2**: My suggestion is based on the OP wanting to know why sampling randomly from the "memory buffer" (replay memory) is effective. This seems the strongest theme in their question. To me, the most salient points are as per my first comment. Namely that the Bellman equation links data from $t$ and $t+1$ time steps and needs no other data from any trajectory, and that neural networks *prefer* iid data which is not correlated (as it would be if you trained strictly in sequence)  
  > **username_3**: @username_2 I see. I tried to answer more of the first question (regarding the function approximation and training on random data) since it is very important in understanding the second question. I'm starting to realize more and more why we try to limit posts to one question.   Also during my time on this board, I've seen that you have some of the best etiquette and do not 'snipe' answers from other people and instead offer comments for slight improvements. Just to be sure, by giving me this suggestion, do you want me to add on to my answer, or would you like to make a separate answer?  
  > **username_2**: Generally, if there is already a good answer that I mostly agree with, then I prefer to offer a few things that IMO could help, instead of writing my own. It's still just my opinion, and it's your answer. If you don't think it improves things on balance, e.g. it over-complicates the answer, then there's no pressure.  
  > **username_0**: Ok guys, thanks for the answers. If I understood right: I interpret @username_3 's answer more as "randomness increase robustness of the network".  While username_2 seems to have understood my question which is, I put it easly, "why is the batch random?". The problem is now, that I didn't understand username_2's answer. Is this the (or part of) the answer? https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables  
  > **username_3**: @username_0  I'll try to incorporate the Bellman equation into my answer and make edits over the next day. In the meantime, if you would reduce the post to contain a single question, it would help me out.  
  > **username_0**: @username_3 I update the original question with more information. And asking only one question  
  > **username_3**: @username_0 I updated my answer. If it isn't satisfactory, please let me know which parts could use more explanation. This answer needs to be high-level, and I can point out other written sources if there are still concerns.  
  > **username_3**: @username_2 See above. I didn't discuss the Bellman equation, but I think my answer is sufficient. That is about as much detail as I can write as an answer on this site without needing the question to be more focused.  
  > **username_2**: It's all solidly correct as far as I can see, and I don't have any further suggestions.  
  
  ---
  
  