Body: <p>How does n-step TD removes the notion of time-step as referenced in Sutton and Barto (2nd edition, Page 163) below?</p>
  <blockquote>
  <p>Another way of looking at the benefits of n-step methods is that they <strong>free you from the tyranny of the time step</strong>. With one-step TD methods the same time step determines how often the action can be changed and the time interval over which bootstrapping is done. With one-step TD methods, these time intervals are the same, and so a compromise must be made. n-step methods enable bootstrapping to occur over multiple steps, freeing us from the tyranny of the single time step.</p>
  </blockquote>
  <p>Consider the following example.</p>
  <p>We know our n-step update equation as: <span class="math-container">$$V_{t+n}(S_t) = V_{t+n-1}(S_t) + \alpha [G_{t:t+n} - V_{t+n-1}(S_t)]$$</span></p>
  <p>Now, let <span class="math-container">$t=0$</span> and <span class="math-container">$n=2$</span>. This gives us: <span class="math-container">$V_2(S_0) = V_1(S_0) + \alpha [G_{0:2} - V_1(S_0)]$</span>.</p>
  <p>Before our n-step TD prediction algorithm starts, we initialize with <span class="math-container">$V_0$</span>. But we use <span class="math-container">$V_1$</span>. Why? And how do we calculate <span class="math-container">$V_1$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-21T10:41:44.127Z
Comments.Text: It seems to me you're asking 2 distinct questions here. "how do we calculate V1?" and "how does n-step TD removes the notion of time-step?". Please, edit your post to leave just one question per post, although the questions are related to the same topic. Ask the second question in a different post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-21T17:20:13.423Z
Comments.Text: Actually, I wanted to use the former question as an example to illustrate actual question I would like to be answered. I edited the question again. If the issue still persists, I'll split the questions into different post.
Comments.UserDisplayName: user529295
Tags: <temporal-difference-methods><model-free-methods>
Title: How does n-step Temporal Difference remove the notion of time-step?
UserDisplayName: user529295
ThreadText: # How does n-step Temporal Difference remove the notion of time-step?
  
  (Asked by: username_0 on 2021-08-18)
  
  How does n-step TD removes the notion of time-step as referenced in Sutton and Barto (2nd edition, Page 163) below?
  
  > Another way of looking at the benefits of n-step methods is that they **free you from the tyranny of the time step**. With one-step TD methods the same time step determines how often the action can be changed and the time interval over which bootstrapping is done. With one-step TD methods, these time intervals are the same, and so a compromise must be made. n-step methods enable bootstrapping to occur over multiple steps, freeing us from the tyranny of the single time step.
  
  Consider the following example.
  
  We know our n-step update equation as: $$V\_{t+n}(S\_t) = V\_{t+n-1}(S\_t) + \alpha [G\_{t:t+n} - V\_{t+n-1}(S\_t)]$$
  
  Now, let $t=0$ and $n=2$. This gives us: $V\_2(S\_0) = V\_1(S\_0) + \alpha [G\_{0:2} - V\_1(S\_0)]$.
  
  Before our n-step TD prediction algorithm starts, we initialize with $V\_0$. But we use $V\_1$. Why? And how do we calculate $V\_1$?
  
  **Tags:** temporal-difference-methods, model-free-methods
  > **Comments:**
  > **username_1**: It seems to me you're asking 2 distinct questions here. "how do we calculate V1?" and "how does n-step TD removes the notion of time-step?". Please, edit your post to leave just one question per post, although the questions are related to the same topic. Ask the second question in a different post.  
  > **username_0**: Actually, I wanted to use the former question as an example to illustrate actual question I would like to be answered. I edited the question again. If the issue still persists, I'll split the questions into different post.  
  
  ---
  
  ## 0 answers
  
  