Answers.Body: <blockquote>
  <p>Why are we choosing more than 1 action in SARSA?</p>
  </blockquote>
  <p>There is never a state where more than one action is chosen.</p>
  <p>The appearance of two <code>Choose</code> statements is an artifact of the loop design and variable management in the pseudocode.</p>
  <blockquote>
  <p>One for going into the next state and the other one for updating the Q function?</p>
  </blockquote>
  <p>Sort of. There is only ever <em>one</em> chosen action for each state.</p>
  <p>However, you need to have two actions in scope - the current action (just taken and observed) and planned next action, in order to process the update rule. Hence there are two variables <code>a</code> and <code>a'</code> and the code needs to generate one or other outside the main loop, or have some other way to ensure that it has access to current and next values. This is also why at the end of the loop, the &quot;next&quot; values <span class="math-container">$s', a'$</span> get copied to the &quot;current&quot; values <span class="math-container">$s, a$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-15T14:37:34.027Z
Answers.LastActivityDate: 2021-12-15T14:52:41.047Z
Answers.LastEditDate: 2021-12-15T14:52:41.047Z
Answers.UserDisplayName: Neil Slater
Body: <p>Here is the pseudocode for SARSA (which I took from <a href="https://ai.stackexchange.com/q/32744/2444">here</a>)</p>
  <p><a href="https://i.sstatic.net/GRFe2.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GRFe2.png" alt="enter image description here" /></a></p>
  <p>Why are we choosing more than 1 action in SARSA? One for going into the next state and the other one for updating the Q function?</p>
  
Tags: <reinforcement-learning><sarsa>
Title: Why are we choosing more than 1 action in SARSA?
UserDisplayName: PeterBe
ThreadText: # Why are we choosing more than 1 action in SARSA?
  
  (Asked by: username_0 on 2021-12-15)
  
  Here is the pseudocode for SARSA (which I took from [here](https://ai.stackexchange.com/q/32744/2444))
  
  [![enter image description here](https://i.sstatic.net/GRFe2.png)](https://i.sstatic.net/GRFe2.png)
  
  Why are we choosing more than 1 action in SARSA? One for going into the next state and the other one for updating the Q function?
  
  **Tags:** reinforcement-learning, sarsa
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-15. Score: 2 | Accepted answer
  
  > Why are we choosing more than 1 action in SARSA?
  
  There is never a state where more than one action is chosen.
  
  The appearance of two `Choose` statements is an artifact of the loop design and variable management in the pseudocode.
  
  > One for going into the next state and the other one for updating the Q function?
  
  Sort of. There is only ever *one* chosen action for each state.
  
  However, you need to have two actions in scope - the current action (just taken and observed) and planned next action, in order to process the update rule. Hence there are two variables `a` and `a'` and the code needs to generate one or other outside the main loop, or have some other way to ensure that it has access to current and next values. This is also why at the end of the loop, the "next" values $s', a'$ get copied to the "current" values $s, a$.
  
  
  ---
  
  