Body: <p>In Sutton &amp; Barto's RL book (<a href="https://incompleteideas.net/book/RLbook2020.pdf#page=171" rel="nofollow noreferrer">page 149</a>) they say:</p>
  <blockquote>
  <p>Sarsa update can be completely replaced by a simple off-policy form</p>
  <p><span class="math-container">$Q_{t+n}(S_t,A_t)=Q_{t+n−1}(S_t,A_t) + \rho_{t+1:t+n} [G_{t:t+n} −  Q_{t+n−1}(S_t,A_t)]$</span></p>
  <p>for <span class="math-container">$0 \leq t &lt; T$</span>. Note that the importance sampling ratio here starts and ends one step later than for n-step TD (7.9). This is because here we are updating a state–action pair. We do not have to care how likely we were to select the action; now that we have selected it we want to learn fully from what happens, with importance sampling only for subsequent actions.</p>
  </blockquote>
  <p>The explanation for importance sampling ratio to start and end one step later is not clear for me. I would be very happy if one can provide a simple and easy-to-understand explanation (a visual explanation through node transitions would be perfect).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-14T09:00:33.233Z
Comments.Text: Prepend the excerpt/text that you're quoting with `>`. This will immediately give an idea what part of your post is a quote. Please, also put your **specific** question in the title, and use more tags other than [tag:reinforcement-learning], to be more specific.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-15T09:54:50.313Z
Comments.Text: @ nbro, all done.
Comments.UserDisplayName: DSPinfinity
Tags: <reinforcement-learning><off-policy-methods><sutton-barto><sarsa><importance-sampling>
Title: Why does importance sampling ratio start and end one step later in off-policy SARSA given in Sutton-Barto's RL book?
UserDisplayName: DSPinfinity
ThreadText: # Why does importance sampling ratio start and end one step later in off-policy SARSA given in Sutton-Barto's RL book?
  
  (Asked by: username_0 on 2022-03-13)
  
  In Sutton & Barto's RL book ([page 149](https://incompleteideas.net/book/RLbook2020.pdf#page=171)) they say:
  
  > Sarsa update can be completely replaced by a simple off-policy form
  >
  > $Q\_{t+n}(S\_t,A\_t)=Q\_{t+n−1}(S\_t,A\_t) + \rho\_{t+1:t+n} [G\_{t:t+n} − Q\_{t+n−1}(S\_t,A\_t)]$
  >
  > for $0 \leq t < T$. Note that the importance sampling ratio here starts and ends one step later than for n-step TD (7.9). This is because here we are updating a state–action pair. We do not have to care how likely we were to select the action; now that we have selected it we want to learn fully from what happens, with importance sampling only for subsequent actions.
  
  The explanation for importance sampling ratio to start and end one step later is not clear for me. I would be very happy if one can provide a simple and easy-to-understand explanation (a visual explanation through node transitions would be perfect).
  
  **Tags:** reinforcement-learning, off-policy-methods, sutton-barto, sarsa, importance-sampling
  > **Comments:**
  > **username_1**: Prepend the excerpt/text that you're quoting with `>`. This will immediately give an idea what part of your post is a quote. Please, also put your **specific** question in the title, and use more tags other than [tag:reinforcement-learning], to be more specific.  
  > **username_0**: @ nbro, all done.  
  
  ---
  
  ## 0 answers
  
  