<p>I was working on an RL problem and I am confused at one specific point. We use <strong>replay memory</strong> so that the network learns about previous actions and how these actions lead to a success or a failure. </p>

<p>Now, to train the neural network, we use batches from this replay or experience memory. But here's my confusion.</p>

<p>Some places like <a href="https://deeplizard.com/learn/video/Bcuj2fTH4_4" rel="nofollow noreferrer">this</a> extract random (non-sequential) batches from the memory to train the neural network but <a href="https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5" rel="nofollow noreferrer">Andrej Karpathy</a> uses the sequential data to train the network. </p>

<p>Can someone tell me why there's the difference?</p>

CC BY-SA 4.0
2019-09-20T10:42:33.963Z
You linked 2 completely different algorithms, that is the reason why there is a difference. One is off-policy and other is on-policy algorithm
Brale
CC BY-SA 4.0
2019-09-20T12:17:39.070Z
What do you mean?
Sarvagya Gupta
CC BY-SA 4.0
2019-09-20T12:49:42.663Z
First link talks about DQN algorithm, second link is implementation of policy gradient algorithm, reinforcement learning isn't only a single algorithm there are multiple kinds of algorithms, you need to read up a bit on it
Brale
CC BY-SA 4.0
2019-09-24T05:08:29.420Z
@Brale_, so correct me if I'm wrong. in DQN, the sequence doesn't matter because we have this Q table that gives an idea to the network what it should do to converge. But in PG, we don't have such a table or any reference hence we have to follow the sequence to see which set of actions led to an optimal policy?
Sarvagya Gupta
<machine-learning><reinforcement-learning><dqn><experience-replay>
What is the difference between random and sequential sampling from the reply memory?
Sarvagya Gupta
# What is the difference between random and sequential sampling from the reply memory?

(Asked by: username_0 on 2019-09-19)

I was working on an RL problem and I am confused at one specific point. We use **replay memory** so that the network learns about previous actions and how these actions lead to a success or a failure.

Now, to train the neural network, we use batches from this replay or experience memory. But here's my confusion.

Some places like [this](https://deeplizard.com/learn/video/Bcuj2fTH4_4) extract random (non-sequential) batches from the memory to train the neural network but [Andrej Karpathy](https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5) uses the sequential data to train the network.

Can someone tell me why there's the difference?

**Tags:** machine-learning, reinforcement-learning, dqn, experience-replay

> **Comments:**
> **username_1**: You linked 2 completely different algorithms, that is the reason why there is a difference. One is off-policy and other is on-policy algorithm  
> **username_0**: What do you mean?  
> **username_1**: First link talks about DQN algorithm, second link is implementation of policy gradient algorithm, reinforcement learning isn't only a single algorithm there are multiple kinds of algorithms, you need to read up a bit on it  
> **username_0**: @username_1_, so correct me if I'm wrong. in DQN, the sequence doesn't matter because we have this Q table that gives an idea to the network what it should do to converge. But in PG, we don't have such a table or any reference hence we have to follow the sequence to see which set of actions led to an optimal policy?  

---

## 0 answers

