Answers.Body: <p>TD(0) is essentially an <em>iterative</em> online algo and your above one-step return TD target is used for TD error to further update value estimation of state <span class="math-container">$s_t$</span> at time step <span class="math-container">$t$</span>, thus the one-step return can only be expressed in terms of value estimation of state <span class="math-container">$s_{t+1}$</span> at the same time step <span class="math-container">$t$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-16T06:53:01.813Z
Answers.LastActivityDate: 2023-03-16T06:53:01.813Z
Answers.UserDisplayName: cinch
Body: <p>in TD(0). for the return we have:</p>
  <p><span class="math-container">$G_{t:t+1}=r_{t+1}+\gamma v_t(s_{t+1})$</span>.</p>
  <p>Why is the time index on right hand side in <span class="math-container">$v$</span> is <span class="math-container">$t$</span>?</p>
  
Tags: <reinforcement-learning><temporal-difference-methods>
Title: Time index in TD(0) return
UserDisplayName: DSPinfinity
ThreadText: # Time index in TD(0) return
  
  (Asked by: username_0 on 2023-03-08)
  
  in TD(0). for the return we have:
  
  $G\_{t:t+1}=r\_{t+1}+\gamma v\_t(s\_{t+1})$.
  
  Why is the time index on right hand side in $v$ is $t$?
  
  **Tags:** reinforcement-learning, temporal-difference-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-16. Score: 1 | Accepted answer
  
  TD(0) is essentially an *iterative* online algo and your above one-step return TD target is used for TD error to further update value estimation of state $s\_t$ at time step $t$, thus the one-step return can only be expressed in terms of value estimation of state $s\_{t+1}$ at the same time step $t$.
  
  
  ---
  
  