<p>Sutton-Barto, second edt, page 128, Exercise 6.7:</p>
<blockquote>
<p>Design an off-policy version of the TD(0) update that can be used with
arbitrary target policy <span class="math-container">$\pi$</span> and covering behavior policy <span class="math-container">$b$</span>, using at each step t the importance sampling ratio <span class="math-container">$\rho_{t:t}$</span> (5.3).</p>
</blockquote>
<p>The answer is:</p>
<p><span class="math-container">$V(S_t) \leftarrow  V(S_t) + \alpha\rho_{t:t}\left[R_{t+1} + V(S_{t+1})-V(S_t)\right]$</span></p>
<p>Normally, if we make a similarity to Monte Carlo method, the target here is <span class="math-container">$R_{t+1} + V(S_{t+1})$</span> and hence I was expecting the answer as:</p>
<p><span class="math-container">$V(S_t) \leftarrow  V(S_t) + \alpha\left[\rho_{t:t}(R_{t+1} + V(S_{t+1}))-V(S_t)\right]$</span></p>
<p>Where is my mistake? Or, is the answer wrong?</p>

CC BY-SA 4.0
2024-04-21T08:32:03.913Z
Where are you getting the official answer from? They are not published in the book.
Neil Slater
CC BY-SA 4.0
2024-04-21T21:12:50.827Z
I have the solutions manual.
DSPinfinity
CC BY-SA 4.0
2024-04-22T06:24:33.177Z
There is more than one place with published solutions. You can even get them from Sutton if you contact him and ask. Please give more information, link if online.
Neil Slater
CC BY-SA 4.0
2024-04-22T13:32:44.137Z
Please, put your specific question in the title. "off-policy TD(0)" is far from being a good title because it could be about 100 things.
nbro
<sutton-barto><temporal-difference-methods>
Derivation of update rule for Off-policy TD(0) with importance sampling ratio
DSPinfinity
# Derivation of update rule for Off-policy TD(0) with importance sampling ratio

(Asked by: username_0 on 2024-04-20)

Sutton-Barto, second edt, page 128, Exercise 6.7:

> Design an off-policy version of the TD(0) update that can be used with
> arbitrary target policy $\pi$ and covering behavior policy $b$, using at each step t the importance sampling ratio $\rho\_{t:t}$ (5.3).

The answer is:

$V(S\_t) \leftarrow V(S\_t) + \alpha\rho\_{t:t}\left[R\_{t+1} + V(S\_{t+1})-V(S\_t)\right]$

Normally, if we make a similarity to Monte Carlo method, the target here is $R\_{t+1} + V(S\_{t+1})$ and hence I was expecting the answer as:

$V(S\_t) \leftarrow V(S\_t) + \alpha\left[\rho\_{t:t}(R\_{t+1} + V(S\_{t+1}))-V(S\_t)\right]$

Where is my mistake? Or, is the answer wrong?

**Tags:** sutton-barto, temporal-difference-methods

> **Comments:**
> **username_1**: Where are you getting the official answer from? They are not published in the book.  
> **username_0**: I have the solutions manual.  
> **username_1**: There is more than one place with published solutions. You can even get them from Sutton if you contact him and ask. Please give more information, link if online.  
> **username_2**: Please, put your specific question in the title. "off-policy TD(0)" is far from being a good title because it could be about 100 things.  

---

## 0 answers

