Answers.Body: <p><span class="math-container">$VT(St)$</span> is the state value function.
  <span class="math-container">$\bar{v}_t$</span> is the average of <span class="math-container">$v_i$</span>, where we make <span class="math-container">$v_i$</span> over a rolling period starting from time <span class="math-container">$t$</span>.
  <span class="math-container">$VT(St)$</span> is a state-value function, for any given time <span class="math-container">$St$</span>, it is not time invariant. Because of initialization, we can say that <span class="math-container">$VT(St') = \bar{v}_t = v_t$</span> if <span class="math-container">$t=St'$</span>. In this sense, it is <span class="math-container">$\text{stationary}$</span>.
  I write <span class="math-container">$VT−1(St)$</span> because I feel it is more close to the true equation (<span class="math-container">$V_t = \sum_i^n \frac{1}{n} RV_i^t$</span> is true), and <span class="math-container">$VT(St)$</span> is in fact a derivation over <span class="math-container">$V_t$</span> with using sample mean.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-17T11:09:11.013Z
Answers.Text: What's worth a downvote, simply a lack of relevant understanding?
Answers.UserDisplayName: Pointwavestandard
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-17T12:32:29.023Z
Answers.Text: There seem to be a lot of errors in this answer, as well as additional variables that you introduce but do not explain. As a result, it is confusing to read and hard to tell what you are trying to say, or whether it answers the question. Example of an error: "any given time $S_t$" - but $S_t$ is *not* a time, it is the state observed at time $t$. It also doesn't help that you are writing $VT$ and $VT - 1$ instead of $V_T$ and $V_{T-1}$
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-17T04:51:22.440Z
Answers.LastActivityDate: 2024-01-17T04:51:22.440Z
Answers.UserDisplayName: Pointwavestandard
Answers.Body: <p>The difference between <span class="math-container">$V_T(S_t)$</span> and <span class="math-container">$V_{T-1}(S_t)$</span> is simply that <span class="math-container">$V_{T-1}(S_t)$</span> is the estimate of <span class="math-container">$V$</span> after <span class="math-container">$T-1$</span> updates. The notation is a bit clumsy since there are two elements of 'time' here -- one being how many updates we have done, which is the subscript of the value function, and the other being the time point in the MDP, which is the subscript of the state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-21T22:52:32.527Z
Answers.Text: it was a wrong picture. Could you please  update your answer based on the new picture?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-21T22:58:16.103Z
Answers.Text: This explanation still tracks with my reading of the new picture.
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-21T22:58:35.717Z
Answers.Text: @foreverska and mine.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-22T07:48:29.283Z
Answers.Text: I think that the previous image also had some incorrect/awkward notation. The author also includes a note that they are being a bit loose with notation. I would note that the notaton does work, sort of, with a single timeline. The value function at time step $T-1$ can be just "the previous value function", as the author doesn't refer any other index values, and updates only occur at the end of episodes.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-22T11:30:19.940Z
Answers.Text: @NeilSlater true, though it would be much clearer if rather than $T-1$ they used something simple like a dash (') to denote the new value function -- similar to how SGD is usually presented.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-21T21:50:09.747Z
Answers.LastActivityDate: 2024-01-21T21:50:09.747Z
Answers.UserDisplayName: David
Body: <p>In state-value function estimation in reinforcement learning,
  I see the following notations: <span class="math-container">$V_T(S_t),\, V_{T-1}(S_t)$</span>. What is the difference between them?</p>
  <p>The context is given below.</p>
  <p><a href="https://i.sstatic.net/Meiob.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Meiob.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-17T08:31:21.450Z
Comments.Text: Please give some context - what was being described when using this notation. A link to where this is written or citing the book and chapter would also be useful. All I can tell you otherwise is that $T$ is an index to multiple state value functions. Plus there is probably an algorithm relating different state value functions. I could *guess* what that is, but much better to see the context
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-21T18:26:08.317Z
Comments.Text: @NeilSlater, I added the context.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-21T19:38:08.150Z
Comments.Text: Thanks. Please could you give a link to the original, as it is an image which will be difficult for some users to read. I don't see the notation $V_{T-1}(S_t)$ anywhere in the image you posted? Could you either post an image with the notation you are asking about, or correct the notation you want to know about in the main question.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-21T22:28:30.513Z
Comments.Text: @NeilSlater, sorry It was the wrong photo. I updated it. There is no link, I had to take a screenshot.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-22T07:28:01.040Z
Comments.Text: A reference to whichever book, course etc this is from then? With page num etc. There is clearly some published source you are referencing.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning>
Title: Confusion in notation for state-value functions in reinforcement learning
UserDisplayName: DSPinfinity
ThreadText: # Confusion in notation for state-value functions in reinforcement learning
  
  (Asked by: username_0 on 2024-01-17)
  
  In state-value function estimation in reinforcement learning,
  I see the following notations: $V\_T(S\_t),\, V\_{T-1}(S\_t)$. What is the difference between them?
  
  The context is given below.
  
  [![enter image description here](https://i.sstatic.net/Meiob.png)](https://i.sstatic.net/Meiob.png)
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: Please give some context - what was being described when using this notation. A link to where this is written or citing the book and chapter would also be useful. All I can tell you otherwise is that $T$ is an index to multiple state value functions. Plus there is probably an algorithm relating different state value functions. I could *guess* what that is, but much better to see the context  
  > **username_0**: @NeilSlater, I added the context.  
  > **username_1**: Thanks. Please could you give a link to the original, as it is an image which will be difficult for some users to read. I don't see the notation $V_{T-1}(S_t)$ anywhere in the image you posted? Could you either post an image with the notation you are asking about, or correct the notation you want to know about in the main question.  
  > **username_0**: @NeilSlater, sorry It was the wrong photo. I updated it. There is no link, I had to take a screenshot.  
  > **username_1**: A reference to whichever book, course etc this is from then? With page num etc. There is clearly some published source you are referencing.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2024-01-21. Score: 4 | Accepted answer
  
  The difference between $V\_T(S\_t)$ and $V\_{T-1}(S\_t)$ is simply that $V\_{T-1}(S\_t)$ is the estimate of $V$ after $T-1$ updates. The notation is a bit clumsy since there are two elements of 'time' here -- one being how many updates we have done, which is the subscript of the value function, and the other being the time point in the MDP, which is the subscript of the state.
  
  > **Comments:**
  > **username_0**: it was a wrong picture. Could you please  update your answer based on the new picture?  
  > **username_4**: This explanation still tracks with my reading of the new picture.  
  > **username_3**: @foreverska and mine.  
  > **username_1**: I think that the previous image also had some incorrect/awkward notation. The author also includes a note that they are being a bit loose with notation. I would note that the notaton does work, sort of, with a single timeline. The value function at time step $T-1$ can be just "the previous value function", as the author doesn't refer any other index values, and updates only occur at the end of episodes.  
  > **username_3**: @NeilSlater true, though it would be much clearer if rather than $T-1$ they used something simple like a dash (') to denote the new value function -- similar to how SGD is usually presented.  
  
  ---
  
  ### Answer by username_2 on 2024-01-17. Score: -3
  
  $VT(St)$ is the state value function.
  $\bar{v}\_t$ is the average of $v\_i$, where we make $v\_i$ over a rolling period starting from time $t$.
  $VT(St)$ is a state-value function, for any given time $St$, it is not time invariant. Because of initialization, we can say that $VT(St') = \bar{v}\_t = v\_t$ if $t=St'$. In this sense, it is $\text{stationary}$.
  I write $VT−1(St)$ because I feel it is more close to the true equation ($V\_t = \sum\_i^n \frac{1}{n} RV\_i^t$ is true), and $VT(St)$ is in fact a derivation over $V\_t$ with using sample mean.
  
  > **Comments:**
  > **username_2**: What's worth a downvote, simply a lack of relevant understanding?  
  > **username_1**: There seem to be a lot of errors in this answer, as well as additional variables that you introduce but do not explain. As a result, it is confusing to read and hard to tell what you are trying to say, or whether it answers the question. Example of an error: "any given time $S_t$" - but $S_t$ is *not* a time, it is the state observed at time $t$. It also doesn't help that you are writing $VT$ and $VT - 1$ instead of $V_T$ and $V_{T-1}$  
  
  ---
  
  