Answers.Body: <p>Expected SARSA can be used either on-policy or off-policy.</p>
  
  <p>The policy that you use in the update step determines which it is. If the update step uses a different weighting for action choices than the policy that actually took the action, then you are using Expected SARSA in an off-policy way.</p>
  
  <p>Q-learning is a special case of Expected SARSA, where the target policy is greedy with respect to the action values, so there is only ever one <span class="math-container">$r_{t+1} + \gamma \text{max}_{a'} Q(s_{t+1}, a')$</span> term to add with a probability <span class="math-container">$1$</span>.</p>
  
  <p>You can also use Expected SARSA, similarly to SARSA, where the behaviour policy and target policy are identical. It is not identical to SARSA though, because it calculates the TD Target over all possible actions <span class="math-container">$r_{t+1} + \gamma \sum_{a'} \pi(a'|s_{t+1}) Q(s_{t+1}, a')$</span></p>
  
  <p>You can construct Expected SARSA updates where <span class="math-container">$\pi(a|s)$</span> is different when selecting which action to explore in the environment (behaviour) and when updating the Q values (target). For instance, you can decide to explore using <span class="math-container">$\epsilon$</span>-greedy with <span class="math-container">$\epsilon=0.1$</span> and update the value function with <span class="math-container">$\epsilon=0.01$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-20T19:09:35.653Z
Answers.Text: Hi Neil, so in the example above, if I was telling the model to update itself using an e-greedy policy targeting actions with highest Q-values, and the policy it used to choose actions was instead completely random, it would then be an off policy approach, correct? I feel like the exercise I specified has a typo of some sort.
Answers.UserDisplayName: Y. Xu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-20T19:10:27.013Z
Answers.Text: The last example you specified has me intrigued. If I coded that up, I would have to write up two e-greedy policy methods, then?
Answers.UserDisplayName: Y. Xu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-20T20:44:26.223Z
Answers.Text: @Y.Xu: That might be one way to do it. Doesn't need to be much code, epsilon is just a parameter of the policy, so you could do it as two instances of the same class for instance.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-20T19:03:17.537Z
Answers.LastActivityDate: 2020-04-20T19:03:17.537Z
Answers.UserDisplayName: Neil Slater
Body: <p>I understand that SARSA is an On-policy algorithm, and Q-learning an off-policy one.
  Sutton and Barto's textbook describes Expected Sarsa thusly:</p>
  
  <blockquote>
    <p>In these cliff walking results Expected Sarsa was used on-policy, but
    in general it might use a policy different from the target policy to
    generate behavior, in which case it becomes an off-policy algorithm.</p>
  </blockquote>
  
  <p>I am fundamentally confused by this - specifically, how do we define when Expected SARSA adopts or disregards policy. The Coursera Course states that it is On-Policy, further confusing me.</p>
  
  <p>My confusions became realized when tackling the Udacity course, specifically a section visualizing Expected SARSA for simple a gridworld (See section 1.11 and 1.12 in link below). Note that the course defines Expected Sarsa as on-policy.
  <a href="https://www.zhenhantom.com/2019/10/27/Deep-Reinforcement-Learning-Part-1/" rel="nofollow noreferrer">https://www.zhenhantom.com/2019/10/27/Deep-Reinforcement-Learning-Part-1/</a></p>
  
  <p>You'll notice the calculation for the new state value Q(s0,a0) as</p>
  
  <blockquote>
    <p>Q(s0, a0) &lt;— 6 + 0.1( -1 + [0.1 x 8] + [0.1 x 7] + [0.7 x 9] + [0.1
    x 8] - 6) = 6.16.</p>
  </blockquote>
  
  <p>This is also the official answer. But this would mean that it is running off policy, given that it is stated that the action taken at S1 corresponds to a shift right, and hence expected SARSA (On policy) should yield you.</p>
  
  <blockquote>
    <p>Q(s0, a0) &lt;— 6 + 0.1( -1 + [0.1 x 8] + [0.1 x 7] + [0.1 x 9] + [0.7 x
    8] - 6) = 6.1</p>
  </blockquote>
  
  <p>The question does state</p>
  
  <blockquote>
    <p>(Suppose that when selecting the actions for the first two timesteps
    in the 100th episode, the agent was following the epsilon-greedy
    policy with respect to the Q-table, with epsilon = 0.4.)</p>
  </blockquote>
  
  <p>But as this same statement existed for the regular SARSA example (which also yields 6.1 as A1 is shift right, as before), I disregarded it.</p>
  
  <p>Any advice is welcome.</p>
  
Tags: <reinforcement-learning><off-policy-methods><sarsa><on-policy-methods><expected-sarsa>
Title: Is Expected SARSA an off-policy or on-policy algorithm?
UserDisplayName: Y. Xu
ThreadText: # Is Expected SARSA an off-policy or on-policy algorithm?
  
  (Asked by: username_0 on 2020-04-20)
  
  I understand that SARSA is an On-policy algorithm, and Q-learning an off-policy one.
  Sutton and Barto's textbook describes Expected Sarsa thusly:
  
  > In these cliff walking results Expected Sarsa was used on-policy, but
  > in general it might use a policy different from the target policy to
  > generate behavior, in which case it becomes an off-policy algorithm.
  
  I am fundamentally confused by this - specifically, how do we define when Expected SARSA adopts or disregards policy. The Coursera Course states that it is On-Policy, further confusing me.
  
  My confusions became realized when tackling the Udacity course, specifically a section visualizing Expected SARSA for simple a gridworld (See section 1.11 and 1.12 in link below). Note that the course defines Expected Sarsa as on-policy.
  <https://www.zhenhantom.com/2019/10/27/Deep-Reinforcement-Learning-Part-1/>
  
  You'll notice the calculation for the new state value Q(s0,a0) as
  
  > Q(s0, a0) <— 6 + 0.1( -1 + [0.1 x 8] + [0.1 x 7] + [0.7 x 9] + [0.1
  > x 8] - 6) = 6.16.
  
  This is also the official answer. But this would mean that it is running off policy, given that it is stated that the action taken at S1 corresponds to a shift right, and hence expected SARSA (On policy) should yield you.
  
  > Q(s0, a0) <— 6 + 0.1( -1 + [0.1 x 8] + [0.1 x 7] + [0.1 x 9] + [0.7 x
  > 8] - 6) = 6.1
  
  The question does state
  
  > (Suppose that when selecting the actions for the first two timesteps
  > in the 100th episode, the agent was following the epsilon-greedy
  > policy with respect to the Q-table, with epsilon = 0.4.)
  
  But as this same statement existed for the regular SARSA example (which also yields 6.1 as A1 is shift right, as before), I disregarded it.
  
  Any advice is welcome.
  
  **Tags:** reinforcement-learning, off-policy-methods, sarsa, on-policy-methods, expected-sarsa
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-20. Score: 3 | Accepted answer
  
  Expected SARSA can be used either on-policy or off-policy.
  
  The policy that you use in the update step determines which it is. If the update step uses a different weighting for action choices than the policy that actually took the action, then you are using Expected SARSA in an off-policy way.
  
  Q-learning is a special case of Expected SARSA, where the target policy is greedy with respect to the action values, so there is only ever one $r\_{t+1} + \gamma \text{max}\_{a'} Q(s\_{t+1}, a')$ term to add with a probability $1$.
  
  You can also use Expected SARSA, similarly to SARSA, where the behaviour policy and target policy are identical. It is not identical to SARSA though, because it calculates the TD Target over all possible actions $r\_{t+1} + \gamma \sum\_{a'} \pi(a'|s\_{t+1}) Q(s\_{t+1}, a')$
  
  You can construct Expected SARSA updates where $\pi(a|s)$ is different when selecting which action to explore in the environment (behaviour) and when updating the Q values (target). For instance, you can decide to explore using $\epsilon$-greedy with $\epsilon=0.1$ and update the value function with $\epsilon=0.01$.
  
  > **Comments:**
  > **username_0**: Hi username_1, so in the example above, if I was telling the model to update itself using an e-greedy policy targeting actions with highest Q-values, and the policy it used to choose actions was instead completely random, it would then be an off policy approach, correct? I feel like the exercise I specified has a typo of some sort.  
  > **username_0**: The last example you specified has me intrigued. If I coded that up, I would have to write up two e-greedy policy methods, then?  
  > **username_1**: @username_0: That might be one way to do it. Doesn't need to be much code, epsilon is just a parameter of the policy, so you could do it as two instances of the same class for instance.  
  
  ---
  
  