Answers.Body: <p>N-step SARSA can be both off policy and on-policy. I think you already know the n step on-policy SARSA. So I am just telling you how n-step SARSA can be off-policy.</p>
  
  <p><strong>Off-policy n-step SARSA:</strong> Now you have two policies, one is target policy, <span class="math-container">$\pi$</span>, (let's say it is a greedy policy), another one is behavior policy, <span class="math-container">$b$</span>, (you are actually following this behavior policy). Since this is off-policy, you do importance sampling for that. So the update rule is like this:</p>
  
  <p><span class="math-container">$$Q_{t+n}(S_{t},A_{t}) = Q_{t+n-1}(S_{t},A_{t}) + \alpha \rho_{t+1:t+n-1}[G_{t:t+n} - Q_{t+n-1}(S_{t},A_{t})],$$</span></p>
  
  <p>where</p>
  
  <p><span class="math-container">$$\rho_{t:h} = \prod_{t=k}^{h} \frac{\pi(A_{k}|S_{k})}{b(A_{k}|S_{k})}$$</span></p>
  
  <p>You are following the behavior policy <span class="math-container">$b$</span>, but shifting the Q values towards the target policy, <span class="math-container">$\pi$</span>. </p>
  
  <p><strong>Off-policy one-step SARSA:</strong> You can think of Q learning as one-step off policy SARSA.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T14:32:18.333Z
Answers.Text: Thanks for your reply. Does that mean on-policy 1-step SARSA is the same as SARSA?
Answers.UserDisplayName: ycenycute
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T15:13:18.507Z
Answers.Text: Yes, one-step on-policy SARSA is called SARSA
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-29T21:11:28.427Z
Answers.Text: Then what is the difference between Q-learning and 1-step off policy SARSA? I understand the algorithms are different. 1-step off policy SARSA uses importance sampling, and Q learning seems just following behavior policy. What would be different results applying these two algorithms? Are they equivalent theoretically?
Answers.UserDisplayName: ycenycute
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-30T07:38:03.863Z
Answers.Text: Q learning and 1step off policy SARA are not exactly the same algorithm. But both are off policy and both follow the behavior policy. But in Q learning you need not consider the importance sampling ratio but in 1 step off policy SARSA you have to know both the behavior policy and target policy (since you have to find the sampling ratio).
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T07:14:09.897Z
Answers.LastActivityDate: 2020-04-27T12:47:07.723Z
Answers.LastEditDate: 2020-04-27T12:47:07.723Z
Answers.UserDisplayName: Swakshar Deb
Body: <p>SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?</p>
  
Tags: <reinforcement-learning><comparison><off-policy-methods><sarsa><on-policy-methods>
Title: What are the differences between 1-step SARSA and SARSA?
UserDisplayName: ycenycute
ThreadText: # What are the differences between 1-step SARSA and SARSA?
  
  (Asked by: username_0 on 2020-04-27)
  
  SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?
  
  **Tags:** reinforcement-learning, comparison, off-policy-methods, sarsa, on-policy-methods
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-27. Score: 0 | Accepted answer
  
  N-step SARSA can be both off policy and on-policy. I think you already know the n step on-policy SARSA. So I am just telling you how n-step SARSA can be off-policy.
  
  **Off-policy n-step SARSA:** Now you have two policies, one is target policy, $\pi$, (let's say it is a greedy policy), another one is behavior policy, $b$, (you are actually following this behavior policy). Since this is off-policy, you do importance sampling for that. So the update rule is like this:
  
  $$Q\_{t+n}(S\_{t},A\_{t}) = Q\_{t+n-1}(S\_{t},A\_{t}) + \alpha \rho\_{t+1:t+n-1}[G\_{t:t+n} - Q\_{t+n-1}(S\_{t},A\_{t})],$$
  
  where
  
  $$\rho\_{t:h} = \prod\_{t=k}^{h} \frac{\pi(A\_{k}|S\_{k})}{b(A\_{k}|S\_{k})}$$
  
  You are following the behavior policy $b$, but shifting the Q values towards the target policy, $\pi$.
  
  **Off-policy one-step SARSA:** You can think of Q learning as one-step off policy SARSA.
  
  > **Comments:**
  > **username_0**: Thanks for your reply. Does that mean on-policy 1-step SARSA is the same as SARSA?  
  > **username_1**: Yes, one-step on-policy SARSA is called SARSA  
  > **username_0**: Then what is the difference between Q-learning and 1-step off policy SARSA? I understand the algorithms are different. 1-step off policy SARSA uses importance sampling, and Q learning seems just following behavior policy. What would be different results applying these two algorithms? Are they equivalent theoretically?  
  > **username_1**: Q learning and 1step off policy SARA are not exactly the same algorithm. But both are off policy and both follow the behavior policy. But in Q learning you need not consider the importance sampling ratio but in 1 step off policy SARSA you have to know both the behavior policy and target policy (since you have to find the sampling ratio).  
  
  ---
  
  