<p>n-step TD(0) is an extension of TD(0) that uses a sequence of forward n rewards and estimated state values to update the value estimate of the current state <span class="math-container">$x_k$</span>, and TD(0) and Monte Carlo prediction are two extreme cases of it. In some reference such as Sutton &amp; Barto’s the subscript is using <span class="math-container">$t$</span> instead of your <span class="math-container">$k$</span> and you can treat it simply as time step of the MDP though nothing prevents you from treating it as iteration number within a loop of an episode. And finally the stochastic approximation iterative equation above is probably not real code implementation thus it has to keep exact time step (k+n) on the LHS.</p>

CC BY-SA 4.0
2023-03-16T06:28:47.900Z
2023-03-16T06:28:47.900Z
cinch
<p>This is n-step TD(0) update rule:</p>
<p><span class="math-container">$v_{k+n}(x_k)=v_{k+n-1}(x_k)+\alpha [g_{k:k+n}-v_{k+n-1}(x_k)]$</span></p>
<p>Why is the subscript on the left hand-side of equation &quot;k+n&quot;, not &quot;k+n-1&quot;? Does the subscript for <span class="math-container">$v$</span> denote time step or iteration number? I am confused.</p>

<reinforcement-learning><temporal-difference-methods>
Confusion in subscript for n-step TD(0)
DSPinfinity
# Confusion in subscript for n-step TD(0)

(Asked by: username_0 on 2023-03-07)

This is n-step TD(0) update rule:

$v\_{k+n}(x\_k)=v\_{k+n-1}(x\_k)+\alpha [g\_{k:k+n}-v\_{k+n-1}(x\_k)]$

Why is the subscript on the left hand-side of equation "k+n", not "k+n-1"? Does the subscript for $v$ denote time step or iteration number? I am confused.

**Tags:** reinforcement-learning, temporal-difference-methods


---

## 1 answers

### Answer by username_1 on 2023-03-16. Score: 1

n-step TD(0) is an extension of TD(0) that uses a sequence of forward n rewards and estimated state values to update the value estimate of the current state $x\_k$, and TD(0) and Monte Carlo prediction are two extreme cases of it. In some reference such as Sutton & Barto’s the subscript is using $t$ instead of your $k$ and you can treat it simply as time step of the MDP though nothing prevents you from treating it as iteration number within a loop of an episode. And finally the stochastic approximation iterative equation above is probably not real code implementation thus it has to keep exact time step (k+n) on the LHS.


---

