Answers.Body: <p>n-step TD(0) is an extension of TD(0) that uses a sequence of forward n rewards and estimated state values to update the value estimate of the current state <span class="math-container">$x_k$</span>, and TD(0) and Monte Carlo prediction are two extreme cases of it. In some reference such as Sutton &amp; Barto’s the subscript is using <span class="math-container">$t$</span> instead of your <span class="math-container">$k$</span> and you can treat it simply as time step of the MDP though nothing prevents you from treating it as iteration number within a loop of an episode. And finally the stochastic approximation iterative equation above is probably not real code implementation thus it has to keep exact time step (k+n) on the LHS.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-16T06:28:47.900Z
Answers.LastActivityDate: 2023-03-16T06:28:47.900Z
Answers.UserDisplayName: cinch
Body: <p>This is n-step TD(0) update rule:</p>
  <p><span class="math-container">$v_{k+n}(x_k)=v_{k+n-1}(x_k)+\alpha [g_{k:k+n}-v_{k+n-1}(x_k)]$</span></p>
  <p>Why is the subscript on the left hand-side of equation &quot;k+n&quot;, not &quot;k+n-1&quot;? Does the subscript for <span class="math-container">$v$</span> denote time step or iteration number? I am confused.</p>
  
Tags: <reinforcement-learning><temporal-difference-methods>
Title: Confusion in subscript for n-step TD(0)
UserDisplayName: DSPinfinity
ThreadText: # Confusion in subscript for n-step TD(0)
  
  (Asked by: username_0 on 2023-03-07)
  
  This is n-step TD(0) update rule:
  
  $v\_{k+n}(x\_k)=v\_{k+n-1}(x\_k)+\alpha [g\_{k:k+n}-v\_{k+n-1}(x\_k)]$
  
  Why is the subscript on the left hand-side of equation "k+n", not "k+n-1"? Does the subscript for $v$ denote time step or iteration number? I am confused.
  
  **Tags:** reinforcement-learning, temporal-difference-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-16. Score: 1
  
  n-step TD(0) is an extension of TD(0) that uses a sequence of forward n rewards and estimated state values to update the value estimate of the current state $x\_k$, and TD(0) and Monte Carlo prediction are two extreme cases of it. In some reference such as Sutton & Barto’s the subscript is using $t$ instead of your $k$ and you can treat it simply as time step of the MDP though nothing prevents you from treating it as iteration number within a loop of an episode. And finally the stochastic approximation iterative equation above is probably not real code implementation thus it has to keep exact time step (k+n) on the LHS.
  
  
  ---
  
  