Answers.Body: <p>In my view, the best way to understand these algorithms is to read the pseudocode (multiple times, if necessary!).</p>
  <p>Here's the pseudocode of <strong>Q-learning</strong>.</p>
  <p><a href="https://i.sstatic.net/rmyhQ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rmyhQ.png" alt="enter image description here" /></a></p>
  <p>Here's the pseudocode of <strong>SARSA</strong>.</p>
  <p><a href="https://i.sstatic.net/tKlR0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tKlR0.png" alt="enter image description here" /></a></p>
  <p>So, as you can see, in SARSA, we choose one action before the episode starts, and, during the episode, we choose (and take) again more actions. In both cases, we choose these actions with the same policy (e.g. <span class="math-container">$\epsilon$</span>-greedy), which is <a href="https://ai.stackexchange.com/q/10492/2444">derived from <span class="math-container">$Q$</span></a>. In Q-learning, we do not choose an action before the episode starts. We only choose and take an action at each step of the episode (like in SARSA). Hence, in SARSA, we choose actions in two places (but only take an action at each step of the episode). Note the difference between <strong>choosing/selecting</strong> an action and <strong>taking</strong> an action in the environment (you may just choose an action to update the Q-function, i.e. without taking it into the environment!).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-14T15:00:33.610Z
Answers.Text: Thanks for your answer nbro. Unfortunately I have to admit that I still have problems understanding it and I have several follow-up questions. 1) Why are we choosing more than 1 action in SARSA? One for going into the next state and the other one for updating the Q function? 2) In SARSA do we only select one action at the very beginning and then we always choose the same action for each step? Does it really make sense to choose the same initally choosen action a regardless of the sate s
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-14T15:00:39.430Z
Answers.Text: 3) Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm.  4) How can we calculate the Q(s', a') in both SARSA and Q-learning for updating the Q-function. After having taken an action a at state s we get the reward r which we can observe. But we cannot observe Q(s',a') from the environment as far as I see it.
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-14T16:28:48.687Z
Answers.Text: @PeterBe Create **one post for each of these specific questions** and I or someone else will answer them. Please, really only one question per post and provide the context for each question in their separate post.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-14T16:32:46.413Z
Answers.Text: Thanks for your answer and effort. I understand that generally each question should be asked in a separate post. However, at least the first 2 questions of mine are direct follow-up questions to your answer. It does not make sense to ask question 1) and 2) in a separate post, because I would exactly ask the same think as in this post (question 1) or react directly to something you mentioned in your answer (question 2).
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-14T16:38:19.783Z
Answers.Text: @PeterBe In this answer, I am answering the question "What is meant by two action selections?" in the context of SARSA. This is actually different than explaining **why** we have 2 action selections in SARSA. If it's a follow-up question or not, it doesn't matter. Your questions could be useful for future readers, that's why I am asking you to ask each of your specific questions in their separate post. More people might have those same questions (but not exactly the same combination of questions: that's why it's important to ask each specific question in a separate post).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-14T16:51:56.103Z
Answers.Text: Thanks for your comment. Actually my first 2 questions were also targeting at "what" are the 2 actions. I still don't understand what they are. As asked in my question 1) is one action for going into the next state and the other one for updating the Q function? and as asked in question 2) what is the first action really about? Is it one action that will not change througout all episodes? What is the first action doing and what is the second action doing? I have problems understanding this from your answer. This is why I think that at least those 2 questions are directly related
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-14T16:59:58.233Z
Answers.Text: @PeterBe I will answer all your questions, if you just ask them in a separate post.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-13T17:16:41.297Z
Answers.LastActivityDate: 2021-12-13T17:32:48.537Z
Answers.LastEditDate: 2021-12-13T17:32:48.537Z
Answers.UserDisplayName: nbro
Body: <p>I have some difficulties understanding the difference between Q-learning and SARSA. Here (<a href="https://ai.stackexchange.com/questions/21910/what-are-the-differences-between-sarsa-and-q-learning">What are the differences between SARSA and Q-learning?</a>) the following updating formulas are given:</p>
  <h3>Q-Learning</h3>
  <p><span class="math-container">$$Q(s,a) = Q(s,a) + \alpha (R_{t+1} + \gamma \max_aQ(s',a) - Q(s,a))$$</span></p>
  <h3>SARSA</h3>
  <p><span class="math-container">$$Q(s,a) = Q(s,a) + \alpha (R_{t+1} + \gamma Q(s',a') - Q(s,a))$$</span></p>
  <p>I know that SARSA is for on-policy learning while Q-learning is off-policy learning. So, in Q-learning, the epsilon-greedy policy (or epsilon-soft or softmax policy) is chosen for selecting the actions and the greedy policy is chosen for updating the Q-values. In SARSA the epsilon-greedy policy (or epsilon-soft or softmax policy) is chosen for selecting the actions and for updating the Q function.</p>
  <p>So, actually, I have a question on that:</p>
  <p>On this website (<a href="https://www.cse.unsw.edu.au/%7Ecs9417ml/RL1/algorithms.html" rel="nofollow noreferrer">https://www.cse.unsw.edu.au/~cs9417ml/RL1/algorithms.html</a>) there is written for SARSA</p>
  <blockquote>
  <p>As you can see, there are two action selection steps needed, for determining the next state-action pair along with the first.</p>
  </blockquote>
  <p>What is meant by two action selections? Normally you can only select one action per iteration. The other &quot;selection&quot; should be for the update.</p>
  
Tags: <reinforcement-learning><comparison><q-learning><sarsa>
Title: What is meant by "two action selections" in SARSA?
UserDisplayName: PeterBe
ThreadText: # What is meant by "two action selections" in SARSA?
  
  (Asked by: username_0 on 2021-12-13)
  
  I have some difficulties understanding the difference between Q-learning and SARSA. Here ([What are the differences between SARSA and Q-learning?](https://ai.stackexchange.com/questions/21910/what-are-the-differences-between-sarsa-and-q-learning)) the following updating formulas are given:
  
  ### Q-Learning
  
  $$Q(s,a) = Q(s,a) + \alpha (R\_{t+1} + \gamma \max\_aQ(s',a) - Q(s,a))$$
  
  ### SARSA
  
  $$Q(s,a) = Q(s,a) + \alpha (R\_{t+1} + \gamma Q(s',a') - Q(s,a))$$
  
  I know that SARSA is for on-policy learning while Q-learning is off-policy learning. So, in Q-learning, the epsilon-greedy policy (or epsilon-soft or softmax policy) is chosen for selecting the actions and the greedy policy is chosen for updating the Q-values. In SARSA the epsilon-greedy policy (or epsilon-soft or softmax policy) is chosen for selecting the actions and for updating the Q function.
  
  So, actually, I have a question on that:
  
  On this website ([https://www.cse.unsw.edu.au/~cs9417ml/RL1/algorithms.html](https://www.cse.unsw.edu.au/%7Ecs9417ml/RL1/algorithms.html)) there is written for SARSA
  
  > As you can see, there are two action selection steps needed, for determining the next state-action pair along with the first.
  
  What is meant by two action selections? Normally you can only select one action per iteration. The other "selection" should be for the update.
  
  **Tags:** reinforcement-learning, comparison, q-learning, sarsa
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-13. Score: 3
  
  In my view, the best way to understand these algorithms is to read the pseudocode (multiple times, if necessary!).
  
  Here's the pseudocode of **Q-learning**.
  
  [![enter image description here](https://i.sstatic.net/rmyhQ.png)](https://i.sstatic.net/rmyhQ.png)
  
  Here's the pseudocode of **SARSA**.
  
  [![enter image description here](https://i.sstatic.net/tKlR0.png)](https://i.sstatic.net/tKlR0.png)
  
  So, as you can see, in SARSA, we choose one action before the episode starts, and, during the episode, we choose (and take) again more actions. In both cases, we choose these actions with the same policy (e.g. $\epsilon$-greedy), which is [derived from $Q$](https://ai.stackexchange.com/q/10492/2444). In Q-learning, we do not choose an action before the episode starts. We only choose and take an action at each step of the episode (like in SARSA). Hence, in SARSA, we choose actions in two places (but only take an action at each step of the episode). Note the difference between **choosing/selecting** an action and **taking** an action in the environment (you may just choose an action to update the Q-function, i.e. without taking it into the environment!).
  
  > **Comments:**
  > **username_0**: Thanks for your answer nbro. Unfortunately I have to admit that I still have problems understanding it and I have several follow-up questions. 1) Why are we choosing more than 1 action in SARSA? One for going into the next state and the other one for updating the Q function? 2) In SARSA do we only select one action at the very beginning and then we always choose the same action for each step? Does it really make sense to choose the same initally choosen action a regardless of the sate s  
  > **username_0**: 3) Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm.  4) How can we calculate the Q(s', a') in both SARSA and Q-learning for updating the Q-function. After having taken an action a at state s we get the reward r which we can observe. But we cannot observe Q(s',a') from the environment as far as I see it.  
  > **username_1**: @PeterBe Create **one post for each of these specific questions** and I or someone else will answer them. Please, really only one question per post and provide the context for each question in their separate post.  
  > **username_0**: Thanks for your answer and effort. I understand that generally each question should be asked in a separate post. However, at least the first 2 questions of mine are direct follow-up questions to your answer. It does not make sense to ask question 1) and 2) in a separate post, because I would exactly ask the same think as in this post (question 1) or react directly to something you mentioned in your answer (question 2).  
  > **username_1**: @PeterBe In this answer, I am answering the question "What is meant by two action selections?" in the context of SARSA. This is actually different than explaining **why** we have 2 action selections in SARSA. If it's a follow-up question or not, it doesn't matter. Your questions could be useful for future readers, that's why I am asking you to ask each of your specific questions in their separate post. More people might have those same questions (but not exactly the same combination of questions: that's why it's important to ask each specific question in a separate post).  
  > **username_0**: Thanks for your comment. Actually my first 2 questions were also targeting at "what" are the 2 actions. I still don't understand what they are. As asked in my question 1) is one action for going into the next state and the other one for updating the Q function? and as asked in question 2) what is the first action really about? Is it one action that will not change througout all episodes? What is the first action doing and what is the second action doing? I have problems understanding this from your answer. This is why I think that at least those 2 questions are directly related  
  > **username_1**: @PeterBe I will answer all your questions, if you just ask them in a separate post.  
  
  ---
  
  