Answers.Body: <blockquote>
    <p>Why is the action selection random with Sarsa? </p>
  </blockquote>
  
  <p>A policy could be stochastic. In the case of SARSA, it is stochastic because of the use of <span class="math-container">$\epsilon$</span>-greedy.</p>
  
  <blockquote>
    <p>Isn't it on-policy and therefore ϵ-greedy?</p>
  </blockquote>
  
  <p>I don't quite understand the question. SARSA is on-policy evaluation with <span class="math-container">$\epsilon$</span>-greedy policy. Q-learning is off-policy evaluation with <span class="math-container">$\epsilon$</span>-greedy policy. <span class="math-container">$\epsilon$</span>-greedy is just a way to turn an action-value function into a policy.</p>
  
  <blockquote>
    <p>Because Expected-Sarsa is off-policy the experience it learns from can be from any policy ... How can Exected-Sarsa learning from such policy be generally better than normal Sarsa learning from an ϵ-greedy policy, especially with the same amount of experience?</p>
  </blockquote>
  
  <p>It is unfair to compare different natures of experiences because off-policy experience contains <em>less</em> useful information. Thus, both SARSA and Expected SARSA should use their own on-policy experience for comparison. </p>
  
  <p>While Expected SARSA update step guarantees to reduce the expected TD error, SARSA could only achieve that in expectation (taking many updates with sufficiently small learning rate). Through this perspective, there is little doubt that Expected SARSA should be better.</p>
  
  <blockquote>
    <p>Probably more general: How can on-policy and off-policy algorithms be compared in such way (e.g. through variance) even though their concepts and assumptions are so different?</p>
  </blockquote>
  
  <p>Same as the previous answer, it is unfair to compare them without the same quality of experiences.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-22T07:24:54.150Z
Answers.Text: I had some big misconceptions there. Thank you for the good clarification!
Answers.UserDisplayName: F.M.F.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-27T10:21:25.043Z
Answers.Text: I would add that Expected SARSA can be used in either on-policy or off-policy approaches, it depends if the policy used to calculate the TD error is the same as that used to select actions (and if you are using some n-step off-policy approach, you may need to add importance sampling or look at the n-step Tree Backup algorithm). Some statements about Expected SARSA in Sutton & Barto make more sense when considering just the on-policy version - especially comparisons with normal SARSA
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-27T09:00:51.383Z
Answers.LastActivityDate: 2019-02-27T09:00:51.383Z
Answers.UserDisplayName: Phizaz
Body: <p>Sutton and Barto state in the 2018-version of "Reinforcement Learning: An Introduction" in the context of Expected SARSA (p. 133) the following sentences:</p>
  
  <blockquote>
    <p>Expected SARSA is more complex computationally than Sarsa but, in return, it eliminates the variance due to the random selection of <span class="math-container">$A_{t+1}$</span>. Given the same amount of experience we might expect it to perform slightly better than Sarsa, and indeed it generally does.</p>
  </blockquote>
  
  <p>I have three questions concerning this statement:</p>
  
  <ol>
  <li>Why is the action selection random with Sarsa? Isn't it on-policy and therefore <span class="math-container">$\epsilon$</span>-greedy?</li>
  <li>Because Expected-Sarsa is off-policy the experience it learns from can be from any policy that at least explores everything in the limit e.g. random action-selection with equal probabilities for every action. How can Exected-Sarsa learning from such policy be <em>generally</em> better than normal Sarsa learning from an <span class="math-container">$\epsilon$</span>-greedy policy, especially with the same amount of experience?</li>
  <li>Probably more general: How can on-policy and off-policy algorithms be compared in such way (e.g. through variance) even though their concepts and assumptions are so different?</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-21T20:16:38.567Z
Comments.Text: I guess that by "random selection" they mean the case when $A_{t+1}$ is actually randomly selected (that is, when you do not select the best action) when you use the $\epsilon$-greedy policy.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-21T20:40:30.640Z
Comments.Text: In this context, I think that by "variance" they mean how much "variation" of some "performance measure" or some metric these methods face when being executed, although I think they could have been more precise and unambiguous. They probably mean something specific, as they illustrate in figure 6.3.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><sutton-barto><sarsa><expected-sarsa>
Title: Expected SARSA vs SARSA in "RL: An Introduction"
UserDisplayName: F.M.F.
ThreadText: # Expected SARSA vs SARSA in "RL: An Introduction"
  
  (Asked by: username_0 on 2019-02-21)
  
  Sutton and Barto state in the 2018-version of "Reinforcement Learning: An Introduction" in the context of Expected SARSA (p. 133) the following sentences:
  
  > Expected SARSA is more complex computationally than Sarsa but, in return, it eliminates the variance due to the random selection of $A\_{t+1}$. Given the same amount of experience we might expect it to perform slightly better than Sarsa, and indeed it generally does.
  
  I have three questions concerning this statement:
  
  1. Why is the action selection random with Sarsa? Isn't it on-policy and therefore $\epsilon$-greedy?
  2. Because Expected-Sarsa is off-policy the experience it learns from can be from any policy that at least explores everything in the limit e.g. random action-selection with equal probabilities for every action. How can Exected-Sarsa learning from such policy be *generally* better than normal Sarsa learning from an $\epsilon$-greedy policy, especially with the same amount of experience?
  3. Probably more general: How can on-policy and off-policy algorithms be compared in such way (e.g. through variance) even though their concepts and assumptions are so different?
  
  **Tags:** reinforcement-learning, sutton-barto, sarsa, expected-sarsa
  > **Comments:**
  > **username_1**: I guess that by "random selection" they mean the case when $A_{t+1}$ is actually randomly selected (that is, when you do not select the best action) when you use the $\epsilon$-greedy policy.  
  > **username_1**: In this context, I think that by "variance" they mean how much "variation" of some "performance measure" or some metric these methods face when being executed, although I think they could have been more precise and unambiguous. They probably mean something specific, as they illustrate in figure 6.3.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-02-27. Score: 4 | Accepted answer
  
  > Why is the action selection random with Sarsa?
  
  A policy could be stochastic. In the case of SARSA, it is stochastic because of the use of $\epsilon$-greedy.
  
  > Isn't it on-policy and therefore ϵ-greedy?
  
  I don't quite understand the question. SARSA is on-policy evaluation with $\epsilon$-greedy policy. Q-learning is off-policy evaluation with $\epsilon$-greedy policy. $\epsilon$-greedy is just a way to turn an action-value function into a policy.
  
  > Because Expected-Sarsa is off-policy the experience it learns from can be from any policy ... How can Exected-Sarsa learning from such policy be generally better than normal Sarsa learning from an ϵ-greedy policy, especially with the same amount of experience?
  
  It is unfair to compare different natures of experiences because off-policy experience contains *less* useful information. Thus, both SARSA and Expected SARSA should use their own on-policy experience for comparison.
  
  While Expected SARSA update step guarantees to reduce the expected TD error, SARSA could only achieve that in expectation (taking many updates with sufficiently small learning rate). Through this perspective, there is little doubt that Expected SARSA should be better.
  
  > Probably more general: How can on-policy and off-policy algorithms be compared in such way (e.g. through variance) even though their concepts and assumptions are so different?
  
  Same as the previous answer, it is unfair to compare them without the same quality of experiences.
  
  > **Comments:**
  > **username_3**: I would add that Expected SARSA can be used in either on-policy or off-policy approaches, it depends if the policy used to calculate the TD error is the same as that used to select actions (and if you are using some n-step off-policy approach, you may need to add importance sampling or look at the n-step Tree Backup algorithm). Some statements about Expected SARSA in Sutton & Barto make more sense when considering just the on-policy version - especially comparisons with normal SARSA  
  > **username_0**: I had some big misconceptions there. Thank you for the good clarification!  
  
  ---
  
  