Answers.Body: <p>The main detail that you are missing is that <span class="math-container">$t$</span> does not represent the &quot;current time step&quot; throughout the loop, but is just a variable giving a reference to a time step that you are processing.</p>
  <p>The first statements in the loop are:</p>
  <blockquote>
  <p>Take an action according to <span class="math-container">$\pi(\cdot,S_t)$</span><br/>
  Obbserve and store the next reward <span class="math-container">$R_{t+1}$</span>, and the next state as <span class="math-container">$S_{t+1}$</span></p>
  </blockquote>
  <p>It is clear that after these two steps that the &quot;current&quot; time step is <span class="math-container">$t+1$</span>. The value of <span class="math-container">$t$</span> is then just a reference to a particular time step that all the maths for the other loops is set up to use.</p>
  <p>This in turn means that because the process has the values of <span class="math-container">$R_{t+1}$</span> and <span class="math-container">$S_{t+1}$</span>, then it is ready to process the estimate for time step <span class="math-container">$\tau = t - n + 1$</span>.</p>
  <p>It would not really help to have <span class="math-container">$t$</span> one higher throughout the loop, because you would still need offsets allowing you to refer multiple time steps within a single loop. Instead of having the odd extra <span class="math-container">$+1$</span> you would see some <span class="math-container">$-1$</span>s instead. E.g. you would need to take an action according to <span class="math-container">$\pi(\cdot,S_{t-1})$</span></p>
  <p>You <em>could</em> swap the sequence of the two &quot;If&quot; blocks and have <span class="math-container">$\tau = t - n$</span> as the first statement.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-12T08:56:26.173Z
Answers.LastActivityDate: 2022-03-12T09:02:47.330Z
Answers.LastEditDate: 2022-03-12T09:02:47.330Z
Answers.UserDisplayName: Neil Slater
Body: <p>If <span class="math-container">$\tau$</span> is the time, whose state’s estimate is being updated, and <span class="math-container">$t$</span> is the current time, then, in n-step TD method, we have <span class="math-container">$t=\tau+n$</span> (because we have to wait n-steps, before we can update). However, in the following pseudocode (Sutton &amp; Barto's RL book, page 144), they have: <span class="math-container">$\tau=t-n+1$</span> which means <span class="math-container">$t=\tau+n-1$</span>. What is wrong in my reasoning?</p>
  <p><a href="https://i.sstatic.net/Wckjb.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Wckjb.png" alt="enter image description here" /></a></p>
  
Tags: <reinforcement-learning><temporal-difference-methods><notation><sutton-barto>
Title: Why $ t=τ+n-1$ instead of $t=τ+n$ in n-step TD?
UserDisplayName: DSPinfinity
ThreadText: # Why $ t=τ+n-1$ instead of $t=τ+n$ in n-step TD?
  
  (Asked by: username_0 on 2022-03-12)
  
  If $\tau$ is the time, whose state’s estimate is being updated, and $t$ is the current time, then, in n-step TD method, we have $t=\tau+n$ (because we have to wait n-steps, before we can update). However, in the following pseudocode (Sutton & Barto's RL book, page 144), they have: $\tau=t-n+1$ which means $t=\tau+n-1$. What is wrong in my reasoning?
  
  [![enter image description here](https://i.sstatic.net/Wckjb.png)](https://i.sstatic.net/Wckjb.png)
  
  **Tags:** reinforcement-learning, temporal-difference-methods, notation, sutton-barto
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-12. Score: 2 | Accepted answer
  
  The main detail that you are missing is that $t$ does not represent the "current time step" throughout the loop, but is just a variable giving a reference to a time step that you are processing.
  
  The first statements in the loop are:
  
  > Take an action according to $\pi(\cdot,S\_t)$  
  > Obbserve and store the next reward $R\_{t+1}$, and the next state as $S\_{t+1}$
  
  It is clear that after these two steps that the "current" time step is $t+1$. The value of $t$ is then just a reference to a particular time step that all the maths for the other loops is set up to use.
  
  This in turn means that because the process has the values of $R\_{t+1}$ and $S\_{t+1}$, then it is ready to process the estimate for time step $\tau = t - n + 1$.
  
  It would not really help to have $t$ one higher throughout the loop, because you would still need offsets allowing you to refer multiple time steps within a single loop. Instead of having the odd extra $+1$ you would see some $-1$s instead. E.g. you would need to take an action according to $\pi(\cdot,S\_{t-1})$
  
  You *could* swap the sequence of the two "If" blocks and have $\tau = t - n$ as the first statement.
  
  
  ---
  
  