<p>I had the same question when I am reading the RL textbook from Sutton Bartol <a href="https://cstheory.stackexchange.com/q/38341/34637">as posted here</a>.</p>

<p><a href="https://i.sstatic.net/coSau.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/coSau.jpg" alt="enter image description here"></a></p>

<p>Why do we update <span class="math-container">$W$</span> with <span class="math-container">$\frac{1}{\mu (A_t | S_t)}$</span> instead of <span class="math-container">$\frac{\pi (A_t | S_t)}{\mu (A_t | S_t)}$</span>? </p>

<p>It seems that, with the updating rule from the textbook, whatever action <span class="math-container">$\mu$</span> decides to choose, we automatically assume that <span class="math-container">$\pi$</span> will choose it with 100% probability. But <span class="math-container">$\pi$</span> is greedy with respect to Q. How does this assumption make sense?   </p>

CC BY-SA 4.0
2020-05-05T17:43:30.970Z
It says there that $\pi$ is a deterministic policy
Brale
CC BY-SA 4.0
2020-05-05T18:15:43.773Z
Ah I see! It could be 0 but in case of 0, loop already breaks before updating W. Should have read the algorithms a few more times. Thank you very much!
roy
<reinforcement-learning><monte-carlo-methods>
Why do we update $W$ with $\frac{1}{\mu (A_t | S_t)}$ instead of $\frac{\pi (A_t | S_t)}{\mu (A_t | S_t)}$ in off-policy Monte Carlo control?
roy
# Why do we update $W$ with $\frac{1}{\mu (A_t | S_t)}$ instead of $\frac{\pi (A_t | S_t)}{\mu (A_t | S_t)}$ in off-policy Monte Carlo control?

(Asked by: username_0 on 2020-05-05)

I had the same question when I am reading the RL textbook from Sutton Bartol [as posted here](https://cstheory.stackexchange.com/q/38341/34637).

[![enter image description here](https://i.sstatic.net/coSau.jpg)](https://i.sstatic.net/coSau.jpg)

Why do we update $W$ with $\frac{1}{\mu (A\_t | S\_t)}$ instead of $\frac{\pi (A\_t | S\_t)}{\mu (A\_t | S\_t)}$?

It seems that, with the updating rule from the textbook, whatever action $\mu$ decides to choose, we automatically assume that $\pi$ will choose it with 100% probability. But $\pi$ is greedy with respect to Q. How does this assumption make sense?

**Tags:** reinforcement-learning, monte-carlo-methods

> **Comments:**
> **username_1**: It says there that $\pi$ is a deterministic policy  
> **username_0**: Ah I see! It could be 0 but in case of 0, loop already breaks before updating W. Should have read the algorithms a few more times. Thank you very much!  

---

## 0 answers

