Answers.Body: <blockquote>
  <p>Do we only select one action at the very beginning and then we always choose the same action for each step?</p>
  </blockquote>
  <p>No.</p>
  <p>The pseudocode is clear on this, by using the word <code>Choose</code> and referencing a policy.</p>
  <p>If you were expected to take the same action again, then the pseudocode already has the previous action in variable <code>a</code>, so it would not need to state anything about making a choice or using a policy.</p>
  <p>The <span class="math-container">$a \leftarrow a'$</span> notation is common way to describe copying values*, so the variable <code>a</code> is changed at the end of each loop.</p>
  <blockquote>
  <p>Does it really make sense to choose the same initially chosen action <span class="math-container">$a$</span> regardless of the state <span class="math-container">$s$</span>?</p>
  </blockquote>
  <p>Not in this case. Some learning algorithms do use a form of &quot;sticky&quot; exploration where a single exploratory action is committed to for multiple time steps. It can be useful in some environments. But not basic SARSA as described in the question.</p>
  <hr />
  <p>* It avoids the ambiguity of <span class="math-container">$=$</span> as assignment or equality operator. You may also see <span class="math-container">$:=$</span> for assignment as an alternative, but for instance Sutton &amp; Barto use <span class="math-container">$\leftarrow$</span> consistently, and a lot of RL literature follows this convention.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-15T14:33:14.593Z
Answers.LastActivityDate: 2021-12-15T15:02:07.217Z
Answers.LastEditDate: 2021-12-15T15:02:07.217Z
Answers.UserDisplayName: Neil Slater
Body: <p>Here is the pseudocode for SARSA (which I took from <a href="https://ai.stackexchange.com/q/32744/2444">here</a>)</p>
  <p><a href="https://i.sstatic.net/GRFe2.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GRFe2.png" alt="enter image description here" /></a></p>
  <p>Do we only select one action at the very beginning and then we always choose the same action for each step? Does it really make sense to choose the same initially chosen action <span class="math-container">$a$</span> regardless of the state <span class="math-container">$s$</span>?</p>
  
Tags: <reinforcement-learning><sarsa>
Title: Are we choosing the same action in every step in SARSA?
UserDisplayName: PeterBe
ThreadText: # Are we choosing the same action in every step in SARSA?
  
  (Asked by: username_0 on 2021-12-15)
  
  Here is the pseudocode for SARSA (which I took from [here](https://ai.stackexchange.com/q/32744/2444))
  
  [![enter image description here](https://i.sstatic.net/GRFe2.png)](https://i.sstatic.net/GRFe2.png)
  
  Do we only select one action at the very beginning and then we always choose the same action for each step? Does it really make sense to choose the same initially chosen action $a$ regardless of the state $s$?
  
  **Tags:** reinforcement-learning, sarsa
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-15. Score: 2 | Accepted answer
  
  > Do we only select one action at the very beginning and then we always choose the same action for each step?
  
  No.
  
  The pseudocode is clear on this, by using the word `Choose` and referencing a policy.
  
  If you were expected to take the same action again, then the pseudocode already has the previous action in variable `a`, so it would not need to state anything about making a choice or using a policy.
  
  The $a \leftarrow a'$ notation is common way to describe copying values\*, so the variable `a` is changed at the end of each loop.
  
  > Does it really make sense to choose the same initially chosen action $a$ regardless of the state $s$?
  
  Not in this case. Some learning algorithms do use a form of "sticky" exploration where a single exploratory action is committed to for multiple time steps. It can be useful in some environments. But not basic SARSA as described in the question.
  
  ---
  
  \* It avoids the ambiguity of $=$ as assignment or equality operator. You may also see $:=$ for assignment as an alternative, but for instance Sutton & Barto use $\leftarrow$ consistently, and a lot of RL literature follows this convention.
  
  
  ---
  
  