<p>TD(0) is essentially an <em>iterative</em> online algo and your above one-step return TD target is used for TD error to further update value estimation of state <span class="math-container">$s_t$</span> at time step <span class="math-container">$t$</span>, thus the one-step return can only be expressed in terms of value estimation of state <span class="math-container">$s_{t+1}$</span> at the same time step <span class="math-container">$t$</span>.</p>

CC BY-SA 4.0
2023-03-16T06:53:01.813Z
2023-03-16T06:53:01.813Z
cinch
<p>in TD(0). for the return we have:</p>
<p><span class="math-container">$G_{t:t+1}=r_{t+1}+\gamma v_t(s_{t+1})$</span>.</p>
<p>Why is the time index on right hand side in <span class="math-container">$v$</span> is <span class="math-container">$t$</span>?</p>

<reinforcement-learning><temporal-difference-methods>
Time index in TD(0) return
DSPinfinity
# Time index in TD(0) return

(Asked by: username_0 on 2023-03-08)

in TD(0). for the return we have:

$G\_{t:t+1}=r\_{t+1}+\gamma v\_t(s\_{t+1})$.

Why is the time index on right hand side in $v$ is $t$?

**Tags:** reinforcement-learning, temporal-difference-methods


---

## 1 answers

### Answer by username_1 on 2023-03-16. Score: 1 | Accepted answer

TD(0) is essentially an *iterative* online algo and your above one-step return TD target is used for TD error to further update value estimation of state $s\_t$ at time step $t$, thus the one-step return can only be expressed in terms of value estimation of state $s\_{t+1}$ at the same time step $t$.


---

