Answers.Body: <p>TD-learning is based on bootstrapping. The TD target <span class="math-container">$R_{t+1} + \gamma V_t(S_{t+1})$</span> describes the immediate reward (random variable), <span class="math-container">$R_{t+1}$</span>, plus the discounted estimated return (starting from the next state <span class="math-container">$S_{t+1}$</span>, which is also a random variable), <span class="math-container">$V_t(S_{t+1})$</span>. The reason why the discounted estimated return of the next state is calculated based on <span class="math-container">$V_t$</span> is that we do not know <span class="math-container">$V_{t+1}$</span> yet, that is at the current iteration of the algorithm.</p>
  <p>What we want to do, is to find the optimal <span class="math-container">$V = V^*$</span>, which would be the same for all timesteps. Then, <span class="math-container">$V^*_t = V^*_{t+1}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-11T23:09:16.447Z
Answers.LastActivityDate: 2022-03-11T23:09:16.447Z
Answers.UserDisplayName: ddaedalus
Body: <p>In one-step  TD updates,  the target is the first reward plus the discounted estimated value of the next state, which we call the one-step return (<a href="https://incompleteideas.net/book/RLbook2020.pdf#page=165" rel="nofollow noreferrer">page 143 of Sutton &amp; Barto</a>):</p>
  <p><span class="math-container">$$
  G_{t:t+1} \triangleq R_{t+1}+\gamma V_{t}(S_{t+1})
  $$</span></p>
  <p>where <span class="math-container">$V_t: \mathcal{S} \to \mathbb{R} $</span> is the estimate at <span class="math-container">$\textbf{time t}$</span> of <span class="math-container">$v_{\pi}$</span>.</p>
  <p>My question is this: Why do we have <span class="math-container">$t$</span> as subscript in <span class="math-container">$V$</span> instead of <span class="math-container">$t+1$</span> in the expression of <span class="math-container">$G_{t:t+1}$</span>? Since we are at time <span class="math-container">$t+1$</span> where state is <span class="math-container">$S_{t+1}$</span>, it seems more logical to have <span class="math-container">$V_{t+1}(S_{t+1})$</span>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-11T08:42:59.263Z
Comments.Text: Can you provide the source of this notation? Where did you find it?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-11T11:01:32.613Z
Comments.Text: @ bro, Sutton-Barto's RL book, page 143. second paragraph.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-11T16:21:09.360Z
Comments.Text: Please, next time, for completeness, provide this info directly into your post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-11T16:28:53.940Z
Comments.Text: I would imagine that here $V_t$ denotes your estimate of $V$ itself at time $t$. So even though we are estimating the value of the next step, it is only our estimate of $V$ at the $t$th iteration of the algorithm.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><temporal-difference-methods><notation><sutton-barto>
Title: Why do we have $t$ as subscript in $V$ instead of $t+1$ in the expression of $G_{t:t+1}$?
UserDisplayName: DSPinfinity
ThreadText: # Why do we have $t$ as subscript in $V$ instead of $t+1$ in the expression of $G_{t:t+1}$?
  
  (Asked by: username_0 on 2022-03-11)
  
  In one-step TD updates, the target is the first reward plus the discounted estimated value of the next state, which we call the one-step return ([page 143 of Sutton & Barto](https://incompleteideas.net/book/RLbook2020.pdf#page=165)):
  
  $$
  G\_{t:t+1} \triangleq R\_{t+1}+\gamma V\_{t}(S\_{t+1})
  $$
  
  where $V\_t: \mathcal{S} \to \mathbb{R} $ is the estimate at $\textbf{time t}$ of $v\_{\pi}$.
  
  My question is this: Why do we have $t$ as subscript in $V$ instead of $t+1$ in the expression of $G\_{t:t+1}$? Since we are at time $t+1$ where state is $S\_{t+1}$, it seems more logical to have $V\_{t+1}(S\_{t+1})$.
  
  **Tags:** reinforcement-learning, temporal-difference-methods, notation, sutton-barto
  
  > **Comments:**
  > **username_1**: Can you provide the source of this notation? Where did you find it?  
  > **username_0**: @ bro, Sutton-Barto's RL book, page 143. second paragraph.  
  > **username_1**: Please, next time, for completeness, provide this info directly into your post.  
  > **username_2**: I would imagine that here $V_t$ denotes your estimate of $V$ itself at time $t$. So even though we are estimating the value of the next step, it is only our estimate of $V$ at the $t$th iteration of the algorithm.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-03-11. Score: 1 | Accepted answer
  
  TD-learning is based on bootstrapping. The TD target $R\_{t+1} + \gamma V\_t(S\_{t+1})$ describes the immediate reward (random variable), $R\_{t+1}$, plus the discounted estimated return (starting from the next state $S\_{t+1}$, which is also a random variable), $V\_t(S\_{t+1})$. The reason why the discounted estimated return of the next state is calculated based on $V\_t$ is that we do not know $V\_{t+1}$ yet, that is at the current iteration of the algorithm.
  
  What we want to do, is to find the optimal $V = V^\*$, which would be the same for all timesteps. Then, $V^\*\_t = V^\*\_{t+1}$.
  
  
  ---
  
  