Answers.Body: <p>In the <em>tabular case</em> (i.e. discrete states), you just look-up <span class="math-container">$V(s_{t+1})$</span> (note that <span class="math-container">$t$</span> is the subscript of the state <span class="math-container">$s$</span>) in the <span class="math-container">$V$</span> table, like <code>V[s[t + 1]]</code>.</p>
  <p>Otherwise, for continuous states, you learn a function approximation (e.g. neural network), say <span class="math-container">$V_\phi$</span>, and so you just predict over the next state <span class="math-container">$s_{t+1}$</span>: <span class="math-container">$\hat{v}_{t+1} = V_\phi(s_{t+1})$</span>.</p>
  <p>In both tabular and NN cases, you have a random initialization (e.g. zero for the table). In TD learning you estimate <span class="math-container">$V(s_t)$</span> from another estimate <span class="math-container">$V(s_{t+1})$</span>, which is called <em>bootstrapping</em> although is biased. In general to get both <span class="math-container">$V(s_t)$</span> and <span class="math-container">$V(s_{t+1})$</span> you either index or predict over <span class="math-container">$s_t$</span> and <span class="math-container">$s_{t+1}$</span>. Indeed, each time you update (or regress, in case of NN) only <span class="math-container">$V(s_t)$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-07T08:59:55.220Z
Answers.LastActivityDate: 2023-06-07T08:59:55.220Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I just start to learn reinforcement learning and confuse with TD error. We calculate temporal difference with</p>
  <blockquote>
  <p>V(t) = V(t) + α[R<sub>t+1</sub> + γV(t+1) - V(t)]</p>
  </blockquote>
  <p>where inside bracket is called TD error.  The main reason we use this because we do not have to wait until the end of trajectory to get V(t) for every state (just wait until reach next state). My question is how to get V(t+1) and V(t). Do we use neural network to predict the value for both every time we calculate TD for each state?</p>
  
Tags: <temporal-difference-methods>
Title: How to calculate TD error?
UserDisplayName: ryan chandra
ThreadText: # How to calculate TD error?
  
  (Asked by: username_0 on 2023-06-07)
  
  I just start to learn reinforcement learning and confuse with TD error. We calculate temporal difference with
  
  > V(t) = V(t) + α[Rt+1 + γV(t+1) - V(t)]
  
  where inside bracket is called TD error. The main reason we use this because we do not have to wait until the end of trajectory to get V(t) for every state (just wait until reach next state). My question is how to get V(t+1) and V(t). Do we use neural network to predict the value for both every time we calculate TD for each state?
  
  **Tags:** temporal-difference-methods
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-07. Score: 1
  
  In the *tabular case* (i.e. discrete states), you just look-up $V(s\_{t+1})$ (note that $t$ is the subscript of the state $s$) in the $V$ table, like `V[s[t + 1]]`.
  
  Otherwise, for continuous states, you learn a function approximation (e.g. neural network), say $V\_\phi$, and so you just predict over the next state $s\_{t+1}$: $\hat{v}\_{t+1} = V\_\phi(s\_{t+1})$.
  
  In both tabular and NN cases, you have a random initialization (e.g. zero for the table). In TD learning you estimate $V(s\_t)$ from another estimate $V(s\_{t+1})$, which is called *bootstrapping* although is biased. In general to get both $V(s\_t)$ and $V(s\_{t+1})$ you either index or predict over $s\_t$ and $s\_{t+1}$. Indeed, each time you update (or regress, in case of NN) only $V(s\_t)$.
  
  
  ---
  
  