Answers.Body: <p>Your mistake is your definition of on-policy. In both cases, we select actions based on our current action value estimates. It's how we select those actions that differentiates the 2 algorithms.</p>
  <hr />
  <p>An algorithm is off-policy if the <strong>target policy</strong> is different from the <strong>exploration/behaviour policy</strong>.</p>
  <p>An algorithm is on-policy if the 2 policies are <strong>the same</strong>.</p>
  <hr />
  <p>I don't like these definitions because they hide details, which are required to understand the concepts. So, let me just explain why these algorithms are what they are.</p>
  <p>In the case of Q-learning, which is off-policy, you try to learn a <strong>greedy policy</strong>, that is, a policy that always selects a greedy action, which is the action that leads to the highest amount of reward. That's why we use <strong>the max operator</strong> in Q-learning. The max operator selects the greedy action value with respect to the current value function estimate. Eventually, that value function estimate may be optimal. In Q-learning, we often also use the <strong><span class="math-container">$\epsilon$</span>-greedy policy</strong> for taking actions in the environment, but you could use any other exploration policy. So, there are 2 different policies. That's why Q-learning is off-policy.</p>
  <p>In SARSA, in your case, we use <span class="math-container">$\epsilon$</span>-greedy both to take actions in the environment and to update the formula.</p>
  <hr />
  <p>Still, it's unclear where the target policy comes into play in these algorithms. Sure, we use the max to select the action reward in Q-learning, or we use the <span class="math-container">$\epsilon$</span>-greedy policy to compute the <strong>target value</strong>, but why using the max operator, for instance, implies that we're learning the greedy policy? Once we have the optimal value function, we can just act greedily wrt it to get an optimal policy, which is greedy (Huh?). So, basically, we compute the target by assuming we would derive a specific policy from the value function and, in the tabular case, it turns out that this approach will find an optimal value function and policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-26T13:14:52.123Z
Answers.LastActivityDate: 2024-04-27T12:40:47.147Z
Answers.LastEditDate: 2024-04-27T12:40:47.147Z
Answers.UserDisplayName: nbro
Body: <p>Here are SARSA and Q-learning from Sutton &amp; Barto.</p>
  <p><a href="https://i.sstatic.net/XxqkR0cg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XxqkR0cg.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/JvOawr2C.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JvOawr2C.png" alt="enter image description here" /></a></p>
  <p>In these given forms, in my opinion, Q-learning is also on-policy because action selection is based on updated Q values. Where is my mistake, if any?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-25T17:35:25.440Z
Comments.Text: I found a very good answer here: https://stats.stackexchange.com/questions/184657/what-is-the-difference-between-off-policy-and-on-policy-learning
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-25T17:54:39.707Z
Comments.Text: Does it fully answer your question or are there future questions?  If it fully answers your question you might think of deleting the question as there's a high chance it gets flagged as duplicate right now.
Comments.UserDisplayName: foreverska
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-25T18:14:16.783Z
Comments.Text: Yes it answers but in my opinion it will be better if we keep this question here because its title is different: it is specific to Q-learning and SARSA.
Comments.UserDisplayName: DSPinfinity
Tags: <reinforcement-learning><q-learning><off-policy-methods><sarsa><on-policy-methods>
Title: It is not clear why SARSA is on-policy but Q-learning off-policy
UserDisplayName: DSPinfinity
ThreadText: # It is not clear why SARSA is on-policy but Q-learning off-policy
  
  (Asked by: username_0 on 2024-04-25)
  
  Here are SARSA and Q-learning from Sutton & Barto.
  
  [![enter image description here](https://i.sstatic.net/XxqkR0cg.png)](https://i.sstatic.net/XxqkR0cg.png)
  
  [![enter image description here](https://i.sstatic.net/JvOawr2C.png)](https://i.sstatic.net/JvOawr2C.png)
  
  In these given forms, in my opinion, Q-learning is also on-policy because action selection is based on updated Q values. Where is my mistake, if any?
  
  **Tags:** reinforcement-learning, q-learning, off-policy-methods, sarsa, on-policy-methods
  
  > **Comments:**
  > **username_0**: I found a very good answer here: https://stats.stackexchange.com/questions/184657/what-is-the-difference-between-off-policy-and-on-policy-learning  
  > **username_1**: Does it fully answer your question or are there future questions?  If it fully answers your question you might think of deleting the question as there's a high chance it gets flagged as duplicate right now.  
  > **username_0**: Yes it answers but in my opinion it will be better if we keep this question here because its title is different: it is specific to Q-learning and SARSA.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-04-26. Score: 2
  
  Your mistake is your definition of on-policy. In both cases, we select actions based on our current action value estimates. It's how we select those actions that differentiates the 2 algorithms.
  
  ---
  
  An algorithm is off-policy if the **target policy** is different from the **exploration/behaviour policy**.
  
  An algorithm is on-policy if the 2 policies are **the same**.
  
  ---
  
  I don't like these definitions because they hide details, which are required to understand the concepts. So, let me just explain why these algorithms are what they are.
  
  In the case of Q-learning, which is off-policy, you try to learn a **greedy policy**, that is, a policy that always selects a greedy action, which is the action that leads to the highest amount of reward. That's why we use **the max operator** in Q-learning. The max operator selects the greedy action value with respect to the current value function estimate. Eventually, that value function estimate may be optimal. In Q-learning, we often also use the **$\epsilon$-greedy policy** for taking actions in the environment, but you could use any other exploration policy. So, there are 2 different policies. That's why Q-learning is off-policy.
  
  In SARSA, in your case, we use $\epsilon$-greedy both to take actions in the environment and to update the formula.
  
  ---
  
  Still, it's unclear where the target policy comes into play in these algorithms. Sure, we use the max to select the action reward in Q-learning, or we use the $\epsilon$-greedy policy to compute the **target value**, but why using the max operator, for instance, implies that we're learning the greedy policy? Once we have the optimal value function, we can just act greedily wrt it to get an optimal policy, which is greedy (Huh?). So, basically, we compute the target by assuming we would derive a specific policy from the value function and, in the tabular case, it turns out that this approach will find an optimal value function and policy.
  
  
  ---
  
  