Answers.Body: <p>No, also for MC Control the importance sampling weights the update, and not just the return</p>
  <p><a href="https://i.sstatic.net/9TqH2YKN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9TqH2YKN.png" alt="enter image description here" /></a></p>
  <p>But the reason I think can be understood intuitively as:</p>
  <ul>
  <li>if you have the same policy as the behavioral, then the updates should be the same, thus the ratio is 1</li>
  <li>if the behavioral policy takes an action that you would have never taken, then the update should be 0 (as you would never go to that part of the MDP), thus the ratio is 0</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-12T23:19:19.660Z
Answers.Text: I updated the question:  In Monte Carlo methods for state values, importance sampling ratio was simply a multiplier for the return.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-12T22:37:54.670Z
Answers.LastActivityDate: 2024-05-12T22:37:54.670Z
Answers.UserDisplayName: Alberto
Body: <p><a href="https://i.sstatic.net/4sXn2XLj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4sXn2XLj.png" alt="enter image description here" /></a></p>
  <p>Why is importance sampling ratio in n-step TD is multiplying error rather than return?
  In Monte Carlo methods for state values, importance sampling ratio was simply a multiplier for the return.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-13T11:05:44.263Z
Comments.Text: can you please post to the part of the book stating the MC update reweighting as you are reporting?
Comments.UserDisplayName: Alberto
Tags: <reinforcement-learning><temporal-difference-methods><off-policy-methods>
Title: Why is importance sampling ratio in n-step TD multiplying error rather than return n-step return?
UserDisplayName: DSPinfinity
ThreadText: # Why is importance sampling ratio in n-step TD multiplying error rather than return n-step return?
  
  (Asked by: username_0 on 2024-05-12)
  
  [![enter image description here](https://i.sstatic.net/4sXn2XLj.png)](https://i.sstatic.net/4sXn2XLj.png)
  
  Why is importance sampling ratio in n-step TD is multiplying error rather than return?
  In Monte Carlo methods for state values, importance sampling ratio was simply a multiplier for the return.
  
  **Tags:** reinforcement-learning, temporal-difference-methods, off-policy-methods
  > **Comments:**
  > **username_1**: can you please post to the part of the book stating the MC update reweighting as you are reporting?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-12. Score: 0
  
  No, also for MC Control the importance sampling weights the update, and not just the return
  
  [![enter image description here](https://i.sstatic.net/9TqH2YKN.png)](https://i.sstatic.net/9TqH2YKN.png)
  
  But the reason I think can be understood intuitively as:
  
  * if you have the same policy as the behavioral, then the updates should be the same, thus the ratio is 1
  * if the behavioral policy takes an action that you would have never taken, then the update should be 0 (as you would never go to that part of the MDP), thus the ratio is 0
  
  > **Comments:**
  > **username_0**: I updated the question:  In Monte Carlo methods for state values, importance sampling ratio was simply a multiplier for the return.  
  
  ---
  
  