<p>Below is the explanation around your stated OpenAI's earlier tutorial for the value function:</p>
<blockquote>
<p>The On-Policy Value Function, <span class="math-container">$V^{\pi}(s)$</span>, which gives the expected return if you start in state <span class="math-container">$s$</span> and always act according to policy <span class="math-container">$\pi$</span></p>
</blockquote>
<p>Therefore there seems no incompatibility issue of notation of <span class="math-container">$s_0$</span> here which is the  initial state of a trajectory <span class="math-container">$\tau$</span>, for any current state <span class="math-container">$s$</span> you can treat it as an initial state of the <em>reward-to-go</em> trajectory <span class="math-container">$\tau$</span> from that point on until the end of its episode as mentioned in your first reference.</p>

CC BY-SA 4.0
2024-01-25T06:17:56.210Z
2024-01-25T06:17:56.210Z
cinch
<p>As far as I understand from <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html" rel="nofollow noreferrer">here</a> (source: OpenAI), the objective function in Policy Gradient is:</p>
<p><span class="math-container">$$J(\pi_{\theta})=E_{\tau\sim\pi_{\theta}}[R(\tau)],$$</span></p>
<p>where <span class="math-container">$R(\tau)=r_0+r_1+...+r_T$</span>, with <span class="math-container">$r_t$</span> being taken from a trajectory <span class="math-container">$\tau = (s_0,a_0,s_1,a_1,...)$</span>. <span class="math-container">$s_0$</span> is defined as the start state, sampled from a distribution <span class="math-container">$\rho_0$</span>, and thus does not depends on the parameter <span class="math-container">$\theta$</span> for a policy <span class="math-container">$\pi$</span>.</p>
<p>However, in the earlier tutorial <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro.html" rel="nofollow noreferrer">here</a> (source: OpenAI), the definition for the value function <span class="math-container">$V^{\pi}$</span> is <span class="math-container">$V^{\pi}(s)=E_{\tau\sim\pi}[R(\tau)|s_0=s]$</span>. Compared with the definition in Sutton &amp; Barto, i.e. <span class="math-container">$V^{\pi}(s)=E_{\pi}[G_t|s_t=s]$</span>.</p>
<p>Can I correctly understand that OpenAI is now defining <span class="math-container">$s_0$</span> as the <strong>current</strong> state? If so, it conflicts with the earlier definition in the policy gradient setting, where <span class="math-container">$s_0$</span> must be the start state, so that its probability does not depend on the policy.</p>

CC BY-SA 4.0
2024-01-18T15:10:01.320Z
the 'correct' objective for policy gradient algorithms is to maximise the value function from the initial state $s_0$. this is why there should also be a second discount factor in the policy gradient, to discount the state distribution. however, in practice, this rarely happens. you can look at a paper 'is the policy gradient a gradient?' for a more in depth discussion on this.
David
CC BY-SA 4.0
2024-01-23T08:27:02.883Z
@David There are more than one valid PG objectives. Episodic and continuing problems will have different definitions of $J$
Neil Slater
CC BY-SA 4.0
2024-01-23T09:50:55.220Z
@NeilSlater sure but this misunderstanding comes up a lot and is mostly due to the lack of second discount factor that is not added in policy gradient algorithms that people will be familiar with. I'm fairly certain there's a question on this site about why there is a second discount factor in the pseudocode of the Sutton and Barto book which is a common source of confusion
David
<reinforcement-learning><definitions><policy-gradients><value-functions><sutton-barto>
Is $s_0$ the current state in policy gradients?
fermented_bean
# Is $s_0$ the current state in policy gradients?

(Asked by: username_0 on 2024-01-18)

As far as I understand from [here](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html) (source: OpenAI), the objective function in Policy Gradient is:

$$J(\pi\_{\theta})=E\_{\tau\sim\pi\_{\theta}}[R(\tau)],$$

where $R(\tau)=r\_0+r\_1+...+r\_T$, with $r\_t$ being taken from a trajectory $\tau = (s\_0,a\_0,s\_1,a\_1,...)$. $s\_0$ is defined as the start state, sampled from a distribution $\rho\_0$, and thus does not depends on the parameter $\theta$ for a policy $\pi$.

However, in the earlier tutorial [here](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html) (source: OpenAI), the definition for the value function $V^{\pi}$ is $V^{\pi}(s)=E\_{\tau\sim\pi}[R(\tau)|s\_0=s]$. Compared with the definition in Sutton & Barto, i.e. $V^{\pi}(s)=E\_{\pi}[G\_t|s\_t=s]$.

Can I correctly understand that OpenAI is now defining $s\_0$ as the **current** state? If so, it conflicts with the earlier definition in the policy gradient setting, where $s\_0$ must be the start state, so that its probability does not depend on the policy.

**Tags:** reinforcement-learning, definitions, policy-gradients, value-functions, sutton-barto

> **Comments:**
> **username_1**: the 'correct' objective for policy gradient algorithms is to maximise the value function from the initial state $s_0$. this is why there should also be a second discount factor in the policy gradient, to discount the state distribution. however, in practice, this rarely happens. you can look at a paper 'is the policy gradient a gradient?' for a more in depth discussion on this.  
> **username_2**: @username_1 There are more than one valid PG objectives. Episodic and continuing problems will have different definitions of $J$  
> **username_1**: @NeilSlater sure but this misunderstanding comes up a lot and is mostly due to the lack of second discount factor that is not added in policy gradient algorithms that people will be familiar with. I'm fairly certain there's a question on this site about why there is a second discount factor in the pseudocode of the Sutton and Barto book which is a common source of confusion  

---

## 1 answers

### Answer by username_3 on 2024-01-25. Score: 0

Below is the explanation around your stated OpenAI's earlier tutorial for the value function:

> The On-Policy Value Function, $V^{\pi}(s)$, which gives the expected return if you start in state $s$ and always act according to policy $\pi$

Therefore there seems no incompatibility issue of notation of $s\_0$ here which is the initial state of a trajectory $\tau$, for any current state $s$ you can treat it as an initial state of the *reward-to-go* trajectory $\tau$ from that point on until the end of its episode as mentioned in your first reference.


---

