Answers.Body: <p>SARSA requires a tuple <span class="math-container">$S,A,R,S',A'$</span> to do an update, where <span class="math-container">$A'$</span> is the action you have taken at state <span class="math-container">$S'$</span>, which means that you can only do the update once you are at state <span class="math-container">$S''$</span>, where instead Q-learning can do the update at step <span class="math-container">$S'$</span></p>
  <p>Indeed, you can see that is a subtle difference in the implementation of the methods, however, for the sake of the exercise, that's the reason</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-25T18:55:56.250Z
Answers.Text: The answer is still not clear and short. Could you try to make it better?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-25T19:40:44.283Z
Answers.Text: Note that SARSA actually does **not** need to have already reached $S''$ before it can do an update for $Q(S, A)$. Just like Q-learning, it can already run its update after reaching $S'$. However, it *does* need to have already determined which action it wants to execute in $S'$. But it doesn't have to have already *executed* that action, only *determined* what it will be.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-25T23:20:24.623Z
Answers.Text: @DennisSoemers yes, what I meant as you are saying in a much better way, is that it need to first decide the action, then update, thus if the update changes the greedy action (as in the example), they differ in what will happen
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-26T07:19:01.290Z
Answers.Text: It "might* help the OP if you showed the difference in the inner loop as pseudocode with the point at which action taken for the loop highlighted. Most notably SARSA will pass the action choice forward to the next iteration where Q learning doesn't need to (although it *could* so the S&B question is referring to a specific implementation detail)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-25T18:48:25.117Z
Answers.LastActivityDate: 2024-04-25T18:48:25.117Z
Answers.UserDisplayName: Alberto
Body: <p>Below is the Exercise 6.12 from Sutto-Barto and its solution (from the solution manual) but I was not able to understand it. I will be happy if one can make it clearer.</p>
  <p><a href="https://i.sstatic.net/m8FmGDsM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/m8FmGDsM.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-26T12:57:52.010Z
Comments.Text: What haven't you understood? Can you be more specific in your questions? Your posts look so lazy and it seems you don't care to even specify what you don't understand.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><q-learning><sutton-barto>
Title: Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as Sarsa?
UserDisplayName: DSPinfinity
ThreadText: # Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as Sarsa?
  
  (Asked by: username_0 on 2024-04-25)
  
  Below is the Exercise 6.12 from Sutto-Barto and its solution (from the solution manual) but I was not able to understand it. I will be happy if one can make it clearer.
  
  [![enter image description here](https://i.sstatic.net/m8FmGDsM.png)](https://i.sstatic.net/m8FmGDsM.png)
  
  **Tags:** reinforcement-learning, q-learning, sutton-barto
  > **Comments:**
  > **username_1**: What haven't you understood? Can you be more specific in your questions? Your posts look so lazy and it seems you don't care to even specify what you don't understand.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-04-25. Score: 2
  
  SARSA requires a tuple $S,A,R,S',A'$ to do an update, where $A'$ is the action you have taken at state $S'$, which means that you can only do the update once you are at state $S''$, where instead Q-learning can do the update at step $S'$
  
  Indeed, you can see that is a subtle difference in the implementation of the methods, however, for the sake of the exercise, that's the reason
  
  > **Comments:**
  > **username_0**: The answer is still not clear and short. Could you try to make it better?  
  > **username_3**: Note that SARSA actually does **not** need to have already reached $S''$ before it can do an update for $Q(S, A)$. Just like Q-learning, it can already run its update after reaching $S'$. However, it *does* need to have already determined which action it wants to execute in $S'$. But it doesn't have to have already *executed* that action, only *determined* what it will be.  
  > **username_2**: @DennisSoemers yes, what I meant as you are saying in a much better way, is that it need to first decide the action, then update, thus if the update changes the greedy action (as in the example), they differ in what will happen  
  > **username_4**: It "might* help the OP if you showed the difference in the inner loop as pseudocode with the point at which action taken for the loop highlighted. Most notably SARSA will pass the action choice forward to the next iteration where Q learning doesn't need to (although it *could* so the S&B question is referring to a specific implementation detail)  
  
  ---
  
  