Body: <p>Sutton-Barto, second edt, page 128, Exercise 6.7:</p>
  <blockquote>
  <p>Design an off-policy version of the TD(0) update that can be used with
  arbitrary target policy <span class="math-container">$\pi$</span> and covering behavior policy <span class="math-container">$b$</span>, using at each step t the importance sampling ratio <span class="math-container">$\rho_{t:t}$</span> (5.3).</p>
  </blockquote>
  <p>The answer is:</p>
  <p><span class="math-container">$V(S_t) \leftarrow  V(S_t) + \alpha\rho_{t:t}\left[R_{t+1} + V(S_{t+1})-V(S_t)\right]$</span></p>
  <p>Normally, if we make a similarity to Monte Carlo method, the target here is <span class="math-container">$R_{t+1} + V(S_{t+1})$</span> and hence I was expecting the answer as:</p>
  <p><span class="math-container">$V(S_t) \leftarrow  V(S_t) + \alpha\left[\rho_{t:t}(R_{t+1} + V(S_{t+1}))-V(S_t)\right]$</span></p>
  <p>Where is my mistake? Or, is the answer wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-21T08:32:03.913Z
Comments.Text: Where are you getting the official answer from? They are not published in the book.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-21T21:12:50.827Z
Comments.Text: I have the solutions manual.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-22T06:24:33.177Z
Comments.Text: There is more than one place with published solutions. You can even get them from Sutton if you contact him and ask. Please give more information, link if online.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-22T13:32:44.137Z
Comments.Text: Please, put your specific question in the title. "off-policy TD(0)" is far from being a good title because it could be about 100 things.
Comments.UserDisplayName: nbro
Tags: <sutton-barto><temporal-difference-methods>
Title: Derivation of update rule for Off-policy TD(0) with importance sampling ratio
UserDisplayName: DSPinfinity
ThreadText: # Derivation of update rule for Off-policy TD(0) with importance sampling ratio
  
  (Asked by: username_0 on 2024-04-20)
  
  Sutton-Barto, second edt, page 128, Exercise 6.7:
  
  > Design an off-policy version of the TD(0) update that can be used with
  > arbitrary target policy $\pi$ and covering behavior policy $b$, using at each step t the importance sampling ratio $\rho\_{t:t}$ (5.3).
  
  The answer is:
  
  $V(S\_t) \leftarrow V(S\_t) + \alpha\rho\_{t:t}\left[R\_{t+1} + V(S\_{t+1})-V(S\_t)\right]$
  
  Normally, if we make a similarity to Monte Carlo method, the target here is $R\_{t+1} + V(S\_{t+1})$ and hence I was expecting the answer as:
  
  $V(S\_t) \leftarrow V(S\_t) + \alpha\left[\rho\_{t:t}(R\_{t+1} + V(S\_{t+1}))-V(S\_t)\right]$
  
  Where is my mistake? Or, is the answer wrong?
  
  **Tags:** sutton-barto, temporal-difference-methods
  > **Comments:**
  > **username_1**: Where are you getting the official answer from? They are not published in the book.  
  > **username_0**: I have the solutions manual.  
  > **username_1**: There is more than one place with published solutions. You can even get them from Sutton if you contact him and ask. Please give more information, link if online.  
  > **username_2**: Please, put your specific question in the title. "off-policy TD(0)" is far from being a good title because it could be about 100 things.  
  
  ---
  
  ## 0 answers
  
  