Body: <p>I was working on an RL problem and I am confused at one specific point. We use <strong>replay memory</strong> so that the network learns about previous actions and how these actions lead to a success or a failure. </p>
  
  <p>Now, to train the neural network, we use batches from this replay or experience memory. But here's my confusion.</p>
  
  <p>Some places like <a href="https://deeplizard.com/learn/video/Bcuj2fTH4_4" rel="nofollow noreferrer">this</a> extract random (non-sequential) batches from the memory to train the neural network but <a href="https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5" rel="nofollow noreferrer">Andrej Karpathy</a> uses the sequential data to train the network. </p>
  
  <p>Can someone tell me why there's the difference?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-20T10:42:33.963Z
Comments.Text: You linked 2 completely different algorithms, that is the reason why there is a difference. One is off-policy and other is on-policy algorithm
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-20T12:17:39.070Z
Comments.Text: What do you mean?
Comments.UserDisplayName: Sarvagya Gupta
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-20T12:49:42.663Z
Comments.Text: First link talks about DQN algorithm, second link is implementation of policy gradient algorithm, reinforcement learning isn't only a single algorithm there are multiple kinds of algorithms, you need to read up a bit on it
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-24T05:08:29.420Z
Comments.Text: @Brale_, so correct me if I'm wrong. in DQN, the sequence doesn't matter because we have this Q table that gives an idea to the network what it should do to converge. But in PG, we don't have such a table or any reference hence we have to follow the sequence to see which set of actions led to an optimal policy?
Comments.UserDisplayName: Sarvagya Gupta
Tags: <machine-learning><reinforcement-learning><dqn><experience-replay>
Title: What is the difference between random and sequential sampling from the reply memory?
UserDisplayName: Sarvagya Gupta
ThreadText: # What is the difference between random and sequential sampling from the reply memory?
  
  (Asked by: username_0 on 2019-09-19)
  
  I was working on an RL problem and I am confused at one specific point. We use **replay memory** so that the network learns about previous actions and how these actions lead to a success or a failure.
  
  Now, to train the neural network, we use batches from this replay or experience memory. But here's my confusion.
  
  Some places like [this](https://deeplizard.com/learn/video/Bcuj2fTH4_4) extract random (non-sequential) batches from the memory to train the neural network but [Andrej Karpathy](https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5) uses the sequential data to train the network.
  
  Can someone tell me why there's the difference?
  
  **Tags:** machine-learning, reinforcement-learning, dqn, experience-replay
  
  > **Comments:**
  > **username_1**: You linked 2 completely different algorithms, that is the reason why there is a difference. One is off-policy and other is on-policy algorithm  
  > **username_0**: What do you mean?  
  > **username_1**: First link talks about DQN algorithm, second link is implementation of policy gradient algorithm, reinforcement learning isn't only a single algorithm there are multiple kinds of algorithms, you need to read up a bit on it  
  > **username_0**: @Brale_, so correct me if I'm wrong. in DQN, the sequence doesn't matter because we have this Q table that gives an idea to the network what it should do to converge. But in PG, we don't have such a table or any reference hence we have to follow the sequence to see which set of actions led to an optimal policy?  
  
  ---
  
  ## 0 answers
  
  