Answers.Body: <p>Below is the explanation around your stated OpenAI's earlier tutorial for the value function:</p>
  <blockquote>
  <p>The On-Policy Value Function, <span class="math-container">$V^{\pi}(s)$</span>, which gives the expected return if you start in state <span class="math-container">$s$</span> and always act according to policy <span class="math-container">$\pi$</span></p>
  </blockquote>
  <p>Therefore there seems no incompatibility issue of notation of <span class="math-container">$s_0$</span> here which is the  initial state of a trajectory <span class="math-container">$\tau$</span>, for any current state <span class="math-container">$s$</span> you can treat it as an initial state of the <em>reward-to-go</em> trajectory <span class="math-container">$\tau$</span> from that point on until the end of its episode as mentioned in your first reference.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-25T06:17:56.210Z
Answers.LastActivityDate: 2024-01-25T06:17:56.210Z
Answers.UserDisplayName: cinch
Body: <p>As far as I understand from <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html" rel="nofollow noreferrer">here</a> (source: OpenAI), the objective function in Policy Gradient is:</p>
  <p><span class="math-container">$$J(\pi_{\theta})=E_{\tau\sim\pi_{\theta}}[R(\tau)],$$</span></p>
  <p>where <span class="math-container">$R(\tau)=r_0+r_1+...+r_T$</span>, with <span class="math-container">$r_t$</span> being taken from a trajectory <span class="math-container">$\tau = (s_0,a_0,s_1,a_1,...)$</span>. <span class="math-container">$s_0$</span> is defined as the start state, sampled from a distribution <span class="math-container">$\rho_0$</span>, and thus does not depends on the parameter <span class="math-container">$\theta$</span> for a policy <span class="math-container">$\pi$</span>.</p>
  <p>However, in the earlier tutorial <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro.html" rel="nofollow noreferrer">here</a> (source: OpenAI), the definition for the value function <span class="math-container">$V^{\pi}$</span> is <span class="math-container">$V^{\pi}(s)=E_{\tau\sim\pi}[R(\tau)|s_0=s]$</span>. Compared with the definition in Sutton &amp; Barto, i.e. <span class="math-container">$V^{\pi}(s)=E_{\pi}[G_t|s_t=s]$</span>.</p>
  <p>Can I correctly understand that OpenAI is now defining <span class="math-container">$s_0$</span> as the <strong>current</strong> state? If so, it conflicts with the earlier definition in the policy gradient setting, where <span class="math-container">$s_0$</span> must be the start state, so that its probability does not depend on the policy.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-18T15:10:01.320Z
Comments.Text: the 'correct' objective for policy gradient algorithms is to maximise the value function from the initial state $s_0$. this is why there should also be a second discount factor in the policy gradient, to discount the state distribution. however, in practice, this rarely happens. you can look at a paper 'is the policy gradient a gradient?' for a more in depth discussion on this.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-23T08:27:02.883Z
Comments.Text: @David There are more than one valid PG objectives. Episodic and continuing problems will have different definitions of $J$
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-23T09:50:55.220Z
Comments.Text: @NeilSlater sure but this misunderstanding comes up a lot and is mostly due to the lack of second discount factor that is not added in policy gradient algorithms that people will be familiar with. I'm fairly certain there's a question on this site about why there is a second discount factor in the pseudocode of the Sutton and Barto book which is a common source of confusion
Comments.UserDisplayName: David
Tags: <reinforcement-learning><definitions><policy-gradients><value-functions><sutton-barto>
Title: Is $s_0$ the current state in policy gradients?
UserDisplayName: fermented_bean
ThreadText: # Is $s_0$ the current state in policy gradients?
  
  (Asked by: username_0 on 2024-01-18)
  
  As far as I understand from [here](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html) (source: OpenAI), the objective function in Policy Gradient is:
  
  $$J(\pi\_{\theta})=E\_{\tau\sim\pi\_{\theta}}[R(\tau)],$$
  
  where $R(\tau)=r\_0+r\_1+...+r\_T$, with $r\_t$ being taken from a trajectory $\tau = (s\_0,a\_0,s\_1,a\_1,...)$. $s\_0$ is defined as the start state, sampled from a distribution $\rho\_0$, and thus does not depends on the parameter $\theta$ for a policy $\pi$.
  
  However, in the earlier tutorial [here](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html) (source: OpenAI), the definition for the value function $V^{\pi}$ is $V^{\pi}(s)=E\_{\tau\sim\pi}[R(\tau)|s\_0=s]$. Compared with the definition in Sutton & Barto, i.e. $V^{\pi}(s)=E\_{\pi}[G\_t|s\_t=s]$.
  
  Can I correctly understand that OpenAI is now defining $s\_0$ as the **current** state? If so, it conflicts with the earlier definition in the policy gradient setting, where $s\_0$ must be the start state, so that its probability does not depend on the policy.
  
  **Tags:** reinforcement-learning, definitions, policy-gradients, value-functions, sutton-barto
  
  > **Comments:**
  > **username_1**: the 'correct' objective for policy gradient algorithms is to maximise the value function from the initial state $s_0$. this is why there should also be a second discount factor in the policy gradient, to discount the state distribution. however, in practice, this rarely happens. you can look at a paper 'is the policy gradient a gradient?' for a more in depth discussion on this.  
  > **username_2**: @username_1 There are more than one valid PG objectives. Episodic and continuing problems will have different definitions of $J$  
  > **username_1**: @username_2 sure but this misunderstanding comes up a lot and is mostly due to the lack of second discount factor that is not added in policy gradient algorithms that people will be familiar with. I'm fairly certain there's a question on this site about why there is a second discount factor in the pseudocode of the Sutton and Barto book which is a common source of confusion  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2024-01-25. Score: 0
  
  Below is the explanation around your stated OpenAI's earlier tutorial for the value function:
  
  > The On-Policy Value Function, $V^{\pi}(s)$, which gives the expected return if you start in state $s$ and always act according to policy $\pi$
  
  Therefore there seems no incompatibility issue of notation of $s\_0$ here which is the initial state of a trajectory $\tau$, for any current state $s$ you can treat it as an initial state of the *reward-to-go* trajectory $\tau$ from that point on until the end of its episode as mentioned in your first reference.
  
  
  ---
  
  