Body: <p>I'm new to RL. Recently, I took a course on Coursera. In the Off-policy MC method, I learned the concept of Importance Sampling as follows:</p>
  <p><a href="https://i.sstatic.net/v0uVd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/v0uVd.png" alt="enter image description here" /></a></p>
  <p>where the importance sampling ratio is the ratio of the target policy over the behavior policy.</p>
  <p>But in Suton book the expectation under the target policy is estimated like this:</p>
  <p><a href="https://i.sstatic.net/lM4yM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lM4yM.png" alt="enter image description here" /></a></p>
  <p>Given that both sources used the same importance sampling ratio. However, I ended up getting <span class="math-container">$E[G_{t}|s] = \sum{G_{t} b \frac{\pi}{\pi}} =  \sum{G_{t} \frac{1}{\rho}\pi} = E[\frac{G_{t}}{\rho_{t:T-1}}|s]$</span> instead</p>
  <p>Did I do something wrong?</p>
  
Tags: <reinforcement-learning><importance-sampling>
Title: Derive Importance Sampling as Expected Value Notation
UserDisplayName: HHD
ThreadText: # Derive Importance Sampling as Expected Value Notation
  
  (Asked by: username_0 on 2021-08-31)
  
  I'm new to RL. Recently, I took a course on Coursera. In the Off-policy MC method, I learned the concept of Importance Sampling as follows:
  
  [![enter image description here](https://i.sstatic.net/v0uVd.png)](https://i.sstatic.net/v0uVd.png)
  
  where the importance sampling ratio is the ratio of the target policy over the behavior policy.
  
  But in Suton book the expectation under the target policy is estimated like this:
  
  [![enter image description here](https://i.sstatic.net/lM4yM.png)](https://i.sstatic.net/lM4yM.png)
  
  Given that both sources used the same importance sampling ratio. However, I ended up getting $E[G\_{t}|s] = \sum{G\_{t} b \frac{\pi}{\pi}} = \sum{G\_{t} \frac{1}{\rho}\pi} = E[\frac{G\_{t}}{\rho\_{t:T-1}}|s]$ instead
  
  Did I do something wrong?
  
  **Tags:** reinforcement-learning, importance-sampling
  
  ---
  
  ## 0 answers
  
  