<p>No, also for MC Control the importance sampling weights the update, and not just the return</p>
<p><a href="https://i.sstatic.net/9TqH2YKN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9TqH2YKN.png" alt="enter image description here" /></a></p>
<p>But the reason I think can be understood intuitively as:</p>
<ul>
<li>if you have the same policy as the behavioral, then the updates should be the same, thus the ratio is 1</li>
<li>if the behavioral policy takes an action that you would have never taken, then the update should be 0 (as you would never go to that part of the MDP), thus the ratio is 0</li>
</ul>

CC BY-SA 4.0
2024-05-12T23:19:19.660Z
I updated the question:  In Monte Carlo methods for state values, importance sampling ratio was simply a multiplier for the return.
DSPinfinity
CC BY-SA 4.0
2024-05-12T22:37:54.670Z
2024-05-12T22:37:54.670Z
Alberto
<p><a href="https://i.sstatic.net/4sXn2XLj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4sXn2XLj.png" alt="enter image description here" /></a></p>
<p>Why is importance sampling ratio in n-step TD is multiplying error rather than return?
In Monte Carlo methods for state values, importance sampling ratio was simply a multiplier for the return.</p>

CC BY-SA 4.0
2024-05-13T11:05:44.263Z
can you please post to the part of the book stating the MC update reweighting as you are reporting?
Alberto
<reinforcement-learning><temporal-difference-methods><off-policy-methods>
Why is importance sampling ratio in n-step TD multiplying error rather than return n-step return?
DSPinfinity
# Why is importance sampling ratio in n-step TD multiplying error rather than return n-step return?

(Asked by: username_0 on 2024-05-12)

[![enter image description here](https://i.sstatic.net/4sXn2XLj.png)](https://i.sstatic.net/4sXn2XLj.png)

Why is importance sampling ratio in n-step TD is multiplying error rather than return?
In Monte Carlo methods for state values, importance sampling ratio was simply a multiplier for the return.

**Tags:** reinforcement-learning, temporal-difference-methods, off-policy-methods

> **Comments:**
> **username_1**: can you please post to the part of the book stating the MC update reweighting as you are reporting?  

---

## 1 answers

### Answer by username_1 on 2024-05-12. Score: 0

No, also for MC Control the importance sampling weights the update, and not just the return

[![enter image description here](https://i.sstatic.net/9TqH2YKN.png)](https://i.sstatic.net/9TqH2YKN.png)

But the reason I think can be understood intuitively as:

* if you have the same policy as the behavioral, then the updates should be the same, thus the ratio is 1
* if the behavioral policy takes an action that you would have never taken, then the update should be 0 (as you would never go to that part of the MDP), thus the ratio is 0

> **Comments:**
> **username_0**: I updated the question:  In Monte Carlo methods for state values, importance sampling ratio was simply a multiplier for the return.  

---

