Answers.Body: <p>For learning, it doesn't matter much how you choose the first action before starting the main loop. That is because the agent doesn't need to learn about transitions <em>to</em> the first state of an episode.</p>
  <p>The thing that does matter is that the first action choice should cover all possible actions with probability greater than zero, in order to guarantee convergence. Conceptually, this is not much different to using exploring starts.</p>
  <p>However, the usual practice is to have just one active policy for SARSA, typically <span class="math-container">$\epsilon$</span>-greedy based on Q values learned so far. It is worth noting here, that as Q values change during learning - which happens on each time step - then the policy may also change (when the greedy action choice changes). So even when you use the same <em>rules</em> to derive the policy in SARSA, the actual policy used may vary, even in the middle of the loop. In that respect, the SARSA algorithm uses many policies, but typically only one approach to determining the current policy.</p>
  <p>If you are using function approximation and also used a very different rule for the policy for the first action, it is possible you could affect the function approximator through sample bias (your training data has different distribution to target data). This is tricky to put a number on in RL, but is usually ignored in off-policy approximation, so should not put you off if you want to try out ideas of using a different first time step policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-16T17:38:41.353Z
Answers.Text: Thanks Neil Slater. Unfortunately I have to admit that I have huge difficulties understanding it and I have several follow-up questions. 1) Why does  "For learning, it doesn't matter much how you choose the first action before" ? Are you here refering to the first action of the whole algorithm or the first action selection in every step as we need to select 2 actions in SARSA in every step 2) Why does the policy in SARSA change when the Q-values are changing? 3) What do you mean by "So even when you use the same rules to derive the policy in SARSA"? What kind of rules are you talking about?
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-16T17:38:58.170Z
Answers.Text: 4) What do you mean by "[SARSA uses] typically only one approach to determining the current policy."? What do you mean by approach here? What is the difference between a policy, rules (Question 3) and approach here? 5) You write "the usual practice is to have just one active policy for SARSA". But is SARSA not an off-policy learning algorithm. So this should mean that it has to use at least 2 policies otherwise it would be on-policy learning as far as I see it.
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-17T07:49:08.227Z
Answers.Text: @PeterBe: I cannot answer your follow-up questions, sorry. You may find it helpful to review basic concepts in RL. I can recommend the book Sutton & Barto, Reinforcement Learning: An Introduction. You can get a copy at http://incompleteideas.net/book/RLbook2020.pdf and I think reading the first 4 or 5 chapters should answer your questions and be a lot more productive for you than more Q&A.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-17T07:56:38.490Z
Answers.Text: Do you know a comprehensive example (other than the book) where it is shown how the algorithm actually works. Any maybe you can try to answer the 2) and the 4) because they confused me most
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-15T14:26:36.463Z
Answers.LastActivityDate: 2021-12-15T17:00:00.513Z
Answers.LastEditDate: 2021-12-15T17:00:00.513Z
Answers.UserDisplayName: Neil Slater
Body: <p>Here is the pseudocode for SARSA (which I took from <a href="https://ai.stackexchange.com/q/32744/2444">here</a>)</p>
  <p><a href="https://i.sstatic.net/GRFe2.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GRFe2.png" alt="enter image description here" /></a></p>
  <p>Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm. But could I, for example, also use different policies in this SARSA framework? For example an <span class="math-container">$\epsilon$</span>-greedy policy and a softmax policy. Maybe the resulting algorithm would not be called SARSA anymore but it would be something similar.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-16T18:08:58.653Z
Comments.Text: I have deleted my answer as it caused too many follow-up questions, thus did not seem to be helping you. I highly recommend reading the book Reinforcement Learning: An Introduction, which covers the rationale behind basic value-based methods very well: http://incompleteideas.net/book/RLbook2020.pdf
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-16T23:36:49.097Z
Comments.Text: @NeilSlater I think you should undelete your answer. It answers the question. If the OP is not able to understand it, it's a different matter.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-17T07:44:38.440Z
Comments.Text: @nbro: OK, I have undeleted the question.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><sarsa><on-policy-methods><exploration-strategies>
Title: Are the two policies in SARSA for choosing an action the same?
UserDisplayName: PeterBe
ThreadText: # Are the two policies in SARSA for choosing an action the same?
  
  (Asked by: username_0 on 2021-12-15)
  
  Here is the pseudocode for SARSA (which I took from [here](https://ai.stackexchange.com/q/32744/2444))
  
  [![enter image description here](https://i.sstatic.net/GRFe2.png)](https://i.sstatic.net/GRFe2.png)
  
  Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm. But could I, for example, also use different policies in this SARSA framework? For example an $\epsilon$-greedy policy and a softmax policy. Maybe the resulting algorithm would not be called SARSA anymore but it would be something similar.
  
  **Tags:** reinforcement-learning, sarsa, on-policy-methods, exploration-strategies
  > **Comments:**
  > **username_1**: I have deleted my answer as it caused too many follow-up questions, thus did not seem to be helping you. I highly recommend reading the book Reinforcement Learning: An Introduction, which covers the rationale behind basic value-based methods very well: http://incompleteideas.net/book/RLbook2020.pdf  
  > **username_2**: @NeilSlater I think you should undelete your answer. It answers the question. If the OP is not able to understand it, it's a different matter.  
  > **username_1**: @nbro: OK, I have undeleted the question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-15. Score: 1
  
  For learning, it doesn't matter much how you choose the first action before starting the main loop. That is because the agent doesn't need to learn about transitions *to* the first state of an episode.
  
  The thing that does matter is that the first action choice should cover all possible actions with probability greater than zero, in order to guarantee convergence. Conceptually, this is not much different to using exploring starts.
  
  However, the usual practice is to have just one active policy for SARSA, typically $\epsilon$-greedy based on Q values learned so far. It is worth noting here, that as Q values change during learning - which happens on each time step - then the policy may also change (when the greedy action choice changes). So even when you use the same *rules* to derive the policy in SARSA, the actual policy used may vary, even in the middle of the loop. In that respect, the SARSA algorithm uses many policies, but typically only one approach to determining the current policy.
  
  If you are using function approximation and also used a very different rule for the policy for the first action, it is possible you could affect the function approximator through sample bias (your training data has different distribution to target data). This is tricky to put a number on in RL, but is usually ignored in off-policy approximation, so should not put you off if you want to try out ideas of using a different first time step policy.
  
  > **Comments:**
  > **username_0**: Thanks Neil Slater. Unfortunately I have to admit that I have huge difficulties understanding it and I have several follow-up questions. 1) Why does  "For learning, it doesn't matter much how you choose the first action before" ? Are you here refering to the first action of the whole algorithm or the first action selection in every step as we need to select 2 actions in SARSA in every step 2) Why does the policy in SARSA change when the Q-values are changing? 3) What do you mean by "So even when you use the same rules to derive the policy in SARSA"? What kind of rules are you talking about?  
  > **username_0**: 4) What do you mean by "[SARSA uses] typically only one approach to determining the current policy."? What do you mean by approach here? What is the difference between a policy, rules (Question 3) and approach here? 5) You write "the usual practice is to have just one active policy for SARSA". But is SARSA not an off-policy learning algorithm. So this should mean that it has to use at least 2 policies otherwise it would be on-policy learning as far as I see it.  
  > **username_1**: @PeterBe: I cannot answer your follow-up questions, sorry. You may find it helpful to review basic concepts in RL. I can recommend the book Sutton & Barto, Reinforcement Learning: An Introduction. You can get a copy at http://incompleteideas.net/book/RLbook2020.pdf and I think reading the first 4 or 5 chapters should answer your questions and be a lot more productive for you than more Q&A.  
  > **username_0**: Do you know a comprehensive example (other than the book) where it is shown how the algorithm actually works. Any maybe you can try to answer the 2) and the 4) because they confused me most  
  
  ---
  
  