<p>N-step SARSA can be both off policy and on-policy. I think you already know the n step on-policy SARSA. So I am just telling you how n-step SARSA can be off-policy.</p>

<p><strong>Off-policy n-step SARSA:</strong> Now you have two policies, one is target policy, <span class="math-container">$\pi$</span>, (let's say it is a greedy policy), another one is behavior policy, <span class="math-container">$b$</span>, (you are actually following this behavior policy). Since this is off-policy, you do importance sampling for that. So the update rule is like this:</p>

<p><span class="math-container">$$Q_{t+n}(S_{t},A_{t}) = Q_{t+n-1}(S_{t},A_{t}) + \alpha \rho_{t+1:t+n-1}[G_{t:t+n} - Q_{t+n-1}(S_{t},A_{t})],$$</span></p>

<p>where</p>

<p><span class="math-container">$$\rho_{t:h} = \prod_{t=k}^{h} \frac{\pi(A_{k}|S_{k})}{b(A_{k}|S_{k})}$$</span></p>

<p>You are following the behavior policy <span class="math-container">$b$</span>, but shifting the Q values towards the target policy, <span class="math-container">$\pi$</span>. </p>

<p><strong>Off-policy one-step SARSA:</strong> You can think of Q learning as one-step off policy SARSA.</p>

CC BY-SA 4.0
2020-04-27T14:32:18.333Z
Thanks for your reply. Does that mean on-policy 1-step SARSA is the same as SARSA?
ycenycute
CC BY-SA 4.0
2020-04-27T15:13:18.507Z
Yes, one-step on-policy SARSA is called SARSA
Swakshar Deb
CC BY-SA 4.0
2020-04-29T21:11:28.427Z
Then what is the difference between Q-learning and 1-step off policy SARSA? I understand the algorithms are different. 1-step off policy SARSA uses importance sampling, and Q learning seems just following behavior policy. What would be different results applying these two algorithms? Are they equivalent theoretically?
ycenycute
CC BY-SA 4.0
2020-04-30T07:38:03.863Z
Q learning and 1step off policy SARA are not exactly the same algorithm. But both are off policy and both follow the behavior policy. But in Q learning you need not consider the importance sampling ratio but in 1 step off policy SARSA you have to know both the behavior policy and target policy (since you have to find the sampling ratio).
Swakshar Deb
CC BY-SA 4.0
2020-04-27T07:14:09.897Z
2020-04-27T12:47:07.723Z
2020-04-27T12:47:07.723Z
Swakshar Deb
<p>SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?</p>

<reinforcement-learning><comparison><off-policy-methods><sarsa><on-policy-methods>
What are the differences between 1-step SARSA and SARSA?
ycenycute
# What are the differences between 1-step SARSA and SARSA?

(Asked by: username_0 on 2020-04-27)

SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?

**Tags:** reinforcement-learning, comparison, off-policy-methods, sarsa, on-policy-methods


---

## 1 answers

### Answer by username_1 on 2020-04-27. Score: 0 | Accepted answer

N-step SARSA can be both off policy and on-policy. I think you already know the n step on-policy SARSA. So I am just telling you how n-step SARSA can be off-policy.

**Off-policy n-step SARSA:** Now you have two policies, one is target policy, $\pi$, (let's say it is a greedy policy), another one is behavior policy, $b$, (you are actually following this behavior policy). Since this is off-policy, you do importance sampling for that. So the update rule is like this:

$$Q\_{t+n}(S\_{t},A\_{t}) = Q\_{t+n-1}(S\_{t},A\_{t}) + \alpha \rho\_{t+1:t+n-1}[G\_{t:t+n} - Q\_{t+n-1}(S\_{t},A\_{t})],$$

where

$$\rho\_{t:h} = \prod\_{t=k}^{h} \frac{\pi(A\_{k}|S\_{k})}{b(A\_{k}|S\_{k})}$$

You are following the behavior policy $b$, but shifting the Q values towards the target policy, $\pi$.

**Off-policy one-step SARSA:** You can think of Q learning as one-step off policy SARSA.

> **Comments:**
> **username_0**: Thanks for your reply. Does that mean on-policy 1-step SARSA is the same as SARSA?  
> **username_1**: Yes, one-step on-policy SARSA is called SARSA  
> **username_0**: Then what is the difference between Q-learning and 1-step off policy SARSA? I understand the algorithms are different. 1-step off policy SARSA uses importance sampling, and Q learning seems just following behavior policy. What would be different results applying these two algorithms? Are they equivalent theoretically?  
> **username_1**: Q learning and 1step off policy SARA are not exactly the same algorithm. But both are off policy and both follow the behavior policy. But in Q learning you need not consider the importance sampling ratio but in 1 step off policy SARSA you have to know both the behavior policy and target policy (since you have to find the sampling ratio).  

---

