Answers.Body: <p>I had the same question too, but finally, I found it was just a mismatch between the explained context and the algorithm given.
  Note that in context, it explains importance sampling ratio for state value function (V) instead of state-action value function (Q) which is the target of the boxed algorithm. Following exercise 5.6, you will know that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-15T04:43:36.747Z
Answers.LastActivityDate: 2024-10-15T04:43:36.747Z
Answers.UserDisplayName: kevinnewgame
Body: <p>I'm studying RL with Sutton and Barto's book. I'd like to ask about the order of execution of a statement in the algorithm below.</p>
  <p><a href="https://i.sstatic.net/ryuYd.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ryuYd.jpg" alt="enter image description here" /></a></p>
  <p>Here, <span class="math-container">$W$</span> (importance sampling ratio) is updated at the end of the <code>For</code> loop.</p>
  <p>But, I think that updating should be located after calculating <span class="math-container">$G$</span> (the return) and before updating <span class="math-container">$C(s,a)$</span> (cumulative of <span class="math-container">$W$</span>). This seems to be right considering the second picture below, which I found in <a href="http://incompleteideas.net/book/first/ebook/node56.html" rel="nofollow noreferrer">http://incompleteideas.net/book/first/ebook/node56.html</a>.</p>
  <p><a href="https://i.sstatic.net/xmJQu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xmJQu.png" alt="enter image description here" /></a></p>
  <p>Is Sutton and Barto's book wrong? Or the two algorithms in Sutton and Barto's book and seconds picture are actually the same, and I am wrong? Is there any difference between the two when implemented? If I am wrong, can you explain the reason?</p>
  
Tags: <reinforcement-learning><monte-carlo-methods><off-policy-methods><importance-sampling>
Title: Should the importance sampling ratio be updated at the end of the for loop in the off-policy Monte Carlo control algorithm?
UserDisplayName: JungYT
ThreadText: # Should the importance sampling ratio be updated at the end of the for loop in the off-policy Monte Carlo control algorithm?
  
  (Asked by: username_0 on 2020-07-07)
  
  I'm studying RL with Sutton and Barto's book. I'd like to ask about the order of execution of a statement in the algorithm below.
  
  [![enter image description here](https://i.sstatic.net/ryuYd.jpg)](https://i.sstatic.net/ryuYd.jpg)
  
  Here, $W$ (importance sampling ratio) is updated at the end of the `For` loop.
  
  But, I think that updating should be located after calculating $G$ (the return) and before updating $C(s,a)$ (cumulative of $W$). This seems to be right considering the second picture below, which I found in <http://incompleteideas.net/book/first/ebook/node56.html>.
  
  [![enter image description here](https://i.sstatic.net/xmJQu.png)](https://i.sstatic.net/xmJQu.png)
  
  Is Sutton and Barto's book wrong? Or the two algorithms in Sutton and Barto's book and seconds picture are actually the same, and I am wrong? Is there any difference between the two when implemented? If I am wrong, can you explain the reason?
  
  **Tags:** reinforcement-learning, monte-carlo-methods, off-policy-methods, importance-sampling
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-10-15. Score: 0
  
  I had the same question too, but finally, I found it was just a mismatch between the explained context and the algorithm given.
  Note that in context, it explains importance sampling ratio for state value function (V) instead of state-action value function (Q) which is the target of the boxed algorithm. Following exercise 5.6, you will know that.
  
  
  ---
  
  