Answers.Body: <p>Let us denote the state we are in at time <span class="math-container">$t$</span> by <span class="math-container">$S_t$</span>. Then at iteration <span class="math-container">$t$</span> we create a placeholder <span class="math-container">$V_{old} = V(S_{t+1})$</span> for the state we will transition into. We then update the value function <span class="math-container">$V(s) \; \forall s \in \mathcal{S}$</span> - i.e. we update the value function for <em>all</em> states in our state space. Let us denote this updated value function by <span class="math-container">$V'(S)$</span>. </p>
  
  <p>At iteration <span class="math-container">$t+1$</span> we calculate <span class="math-container">$\Delta = V'(S_{t+1}) - V_{old} = V'(S_{t+1}) - V(S_{t+1})$</span>, which does not necessarily equal 0 because the placeholder <span class="math-container">$V_{old}$</span> was created using the value function before the last update. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-04T14:46:23.077Z
Answers.Text: Ah, I see! I missed the fact that when TD updates, it updates the whole state space! Thank you so much, David!
Answers.UserDisplayName: roy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-03T16:21:45.447Z
Answers.LastActivityDate: 2020-06-03T18:41:12.397Z
Answers.LastEditDate: 2020-06-03T18:41:12.397Z
Answers.UserDisplayName: David
Body: <p>In the RL textbook by Sutton &amp; Barto section 7.4, the author talked about the "True online TD(<span class="math-container">$\lambda$</span>)". The figure (7.10 in the book) below shows the algorithm. </p>
  
  <p>At the end of each step, <span class="math-container">$V_{old} \leftarrow V(S')$</span> and also <span class="math-container">$S \leftarrow S'$</span>. When we jump to next step, <span class="math-container">$\Delta \leftarrow V(S') - V(S')$</span>, which is 0. It seems that <span class="math-container">$\Delta$</span> is always going to be 0 after step 1. If that is true, it does not make any sense to me. Can you please elaborate on how  <span class="math-container">$\Delta$</span> is updated? </p>
  
  <p><a href="https://i.sstatic.net/vxG22.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vxG22.png" alt="enter image description here"></a></p>
  
Tags: <reinforcement-learning><temporal-difference-methods><td-lambda>
Title: How is $\Delta$ updated in true online TD($\lambda$)?
UserDisplayName: roy
ThreadText: # How is $\Delta$ updated in true online TD($\lambda$)?
  
  (Asked by: username_0 on 2020-06-03)
  
  In the RL textbook by Sutton & Barto section 7.4, the author talked about the "True online TD($\lambda$)". The figure (7.10 in the book) below shows the algorithm.
  
  At the end of each step, $V\_{old} \leftarrow V(S')$ and also $S \leftarrow S'$. When we jump to next step, $\Delta \leftarrow V(S') - V(S')$, which is 0. It seems that $\Delta$ is always going to be 0 after step 1. If that is true, it does not make any sense to me. Can you please elaborate on how $\Delta$ is updated?
  
  [![enter image description here](https://i.sstatic.net/vxG22.png)](https://i.sstatic.net/vxG22.png)
  
  **Tags:** reinforcement-learning, temporal-difference-methods, td-lambda
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-03. Score: 4 | Accepted answer
  
  Let us denote the state we are in at time $t$ by $S\_t$. Then at iteration $t$ we create a placeholder $V\_{old} = V(S\_{t+1})$ for the state we will transition into. We then update the value function $V(s) \; \forall s \in \mathcal{S}$ - i.e. we update the value function for *all* states in our state space. Let us denote this updated value function by $V'(S)$.
  
  At iteration $t+1$ we calculate $\Delta = V'(S\_{t+1}) - V\_{old} = V'(S\_{t+1}) - V(S\_{t+1})$, which does not necessarily equal 0 because the placeholder $V\_{old}$ was created using the value function before the last update.
  
  > **Comments:**
  > **username_0**: Ah, I see! I missed the fact that when TD updates, it updates the whole state space! Thank you so much, David!  
  
  ---
  
  