Body: <p>I had the same question when I am reading the RL textbook from Sutton Bartol <a href="https://cstheory.stackexchange.com/q/38341/34637">as posted here</a>.</p>
  
  <p><a href="https://i.sstatic.net/coSau.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/coSau.jpg" alt="enter image description here"></a></p>
  
  <p>Why do we update <span class="math-container">$W$</span> with <span class="math-container">$\frac{1}{\mu (A_t | S_t)}$</span> instead of <span class="math-container">$\frac{\pi (A_t | S_t)}{\mu (A_t | S_t)}$</span>? </p>
  
  <p>It seems that, with the updating rule from the textbook, whatever action <span class="math-container">$\mu$</span> decides to choose, we automatically assume that <span class="math-container">$\pi$</span> will choose it with 100% probability. But <span class="math-container">$\pi$</span> is greedy with respect to Q. How does this assumption make sense?   </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-05T17:43:30.970Z
Comments.Text: It says there that $\pi$ is a deterministic policy
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-05T18:15:43.773Z
Comments.Text: Ah I see! It could be 0 but in case of 0, loop already breaks before updating W. Should have read the algorithms a few more times. Thank you very much!
Comments.UserDisplayName: roy
Tags: <reinforcement-learning><monte-carlo-methods>
Title: Why do we update $W$ with $\frac{1}{\mu (A_t | S_t)}$ instead of $\frac{\pi (A_t | S_t)}{\mu (A_t | S_t)}$ in off-policy Monte Carlo control?
UserDisplayName: roy
ThreadText: # Why do we update $W$ with $\frac{1}{\mu (A_t | S_t)}$ instead of $\frac{\pi (A_t | S_t)}{\mu (A_t | S_t)}$ in off-policy Monte Carlo control?
  
  (Asked by: username_0 on 2020-05-05)
  
  I had the same question when I am reading the RL textbook from Sutton Bartol [as posted here](https://cstheory.stackexchange.com/q/38341/34637).
  
  [![enter image description here](https://i.sstatic.net/coSau.jpg)](https://i.sstatic.net/coSau.jpg)
  
  Why do we update $W$ with $\frac{1}{\mu (A\_t | S\_t)}$ instead of $\frac{\pi (A\_t | S\_t)}{\mu (A\_t | S\_t)}$?
  
  It seems that, with the updating rule from the textbook, whatever action $\mu$ decides to choose, we automatically assume that $\pi$ will choose it with 100% probability. But $\pi$ is greedy with respect to Q. How does this assumption make sense?
  
  **Tags:** reinforcement-learning, monte-carlo-methods
  > **Comments:**
  > **username_1**: It says there that $\pi$ is a deterministic policy  
  > **username_0**: Ah I see! It could be 0 but in case of 0, loop already breaks before updating W. Should have read the algorithms a few more times. Thank you very much!  
  
  ---
  
  ## 0 answers
  
  