Answers.Body: <blockquote>
    <p>So is the baseline expected to be used in the next iteration when our policy has <em>changed</em>?</p>
  </blockquote>
  
  <p>Yes.</p>
  
  <blockquote>
    <p>To compute the advantage we subtract the state value <span class="math-container">$V(s_{t})$</span> from the action value <span class="math-container">$Q(s_{t},a_{t})$</span>, under the <em>same</em> policy, then why is the old baseline used here in advantage estimation?</p>
  </blockquote>
  
  <p>The precise value of the baseline is not that important. What is important is that the baseline does not depend on the action choice, <span class="math-container">$a$</span>, so it does not impact the gradient estimations or update steps for the policy function you are trying to improve. </p>
  
  <p>You could in theory use a fixed offset instead of <span class="math-container">$V(s)$</span>, or any arbitrary function that does not depend on <span class="math-container">$a$</span>. In some settings the average reward <span class="math-container">$\bar{R}$</span> seen so far is used. </p>
  
  <p>Using a rough approximation to <span class="math-container">$V(s)$</span> - and thus an approximate advantage function overall is useful as it removes a large source of variance in gradient estimates (the inherent value of the current state under the current policy, which is irrelevant to search for <em>adjustments</em> to that  policy). The more accurate <span class="math-container">$V(s)$</span> is, the lower variance, thus faster and more reliable convergence, so you do want it to be a good estimate. But a little bit of lag behind policy updates is acceptable and does not break the algorithm.</p>
  
  <p>For more on this, see <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">Sutton &amp; Barto, chapter 13, section 13.4</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-22T14:23:22.427Z
Answers.LastActivityDate: 2020-05-22T14:23:22.427Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>To be honest I was also struggling with this question for quite some time and I think that it was written this way mostly because of historic reasons. Once you have the advantage <span class="math-container">$A_t = \sum_{i=t}^{T} r_{i+1} - V_\phi(s)$</span>, you can use it directly to update both the value and the policy networks:</p>
  <p><span class="math-container">\begin{align}
  &amp;w = w + \alpha A_t \nabla V_\phi(s) \\
  &amp;\theta = \theta + \alpha A_t \nabla \log \pi_\theta(a|s) 
  \end{align}</span></p>
  <p>To see why this is true note that you update the value network using the squared-error loss <span class="math-container">$L^{VF} = 0.5(\sum r_{t+1} - V_\phi(s))^2$</span>, and when you differentiate this you get simply <span class="math-container">$A_t$</span>. And the loss for the policy network is simply <span class="math-container">$L^{PG} = \log \pi_\theta(a|s) A_t$</span>.</p>
  <p>Essentially you evaluate the value network only one time, instead of two times, which probably back at the time when this algorithm was invented (e.g. in the 1990s) was a big deal.</p>
  <p>If I were to implement vpg with baseline, I would first update the value network and only then calculate the advantage to update the policy network, i.e. I would reshuffle the steps so that I use the updated baseline. However, I would never use vpg with baseline. I don't think anyone has ever used it for something practical and I don't know of any papers that show SotA results with it.</p>
  <p>Note that if you have the value network, instead of using it merely as a baseline, you would be much better off using it to bootstrap the estimate of the return, i.e. <span class="math-container">$A_t = r_{t+1} + V(s_{t+1}) - V(s_t).$</span><br />
  If you take a look at the pseudocode for this algorithm you can see that, again, people update the value network after they compute the advantage. Here, you could again update the value network before that, but now you have to be extra careful. When computing the advantage you have to use the old value network for bootstrapping and the new value network only for the baseline: <span class="math-container">$A_t = r_{t+1} + V_{old}(s_{t+1}) - V_{new}(s_t).$</span> A very brief explanation for this is given at the end of page 55 in John Schulman's phd thesis <a href="http://joschu.net/docs/thesis.pdf" rel="nofollow noreferrer">that you shared</a>.
  So in this case, I would just go on and use the old network and not overthink it.</p>
  <p>Even better than a one-step bootstrap, you could do an n-step bootstrap: <span class="math-container">$A_t^{(n)} = r_{t+1} + r_{t+2} + \cdots + r_{t+n} + V(s_{t+n}) - V(s_t)$</span> and then use a generalized advantage estimation <span class="math-container">$A^{GAE} = \sum_{n=1}^{\inf} (1-\lambda) \lambda ^ {n-1} A_{t}^{(n)}$</span>. When computing <span class="math-container">$A^{GAE}$</span> the value network is used both for bootstrapping and as a baseline and it is not possible to distinguish between the two uses because they are cancelled out in a telescoping sum. This means that the only correct way would be to update the value network after you compute the advantage.</p>
  <p>So you can see that once you have the value network you get from a poor algorithm to state-of-the-art with one simple equation. See <a href="https://pi-tau.github.io/posts/actor-critic/" rel="nofollow noreferrer">here</a> if you want to read more.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-26T23:49:14.700Z
Answers.LastActivityDate: 2023-08-28T09:01:38.463Z
Answers.LastEditDate: 2023-08-28T09:01:38.463Z
Answers.UserDisplayName: pi-tau
Body: <p>Vanilla policy gradient algorithm (using baseline to reduce variance) acc to <a href="http://joschu.net/docs/thesis.pdf" rel="nofollow noreferrer">here</a> (page 16)</p>
  
  <blockquote>
    <p>Initialize policy parameter θ, baseline b</p>
    
    <p>for iteration=1, 2, . . . do</p>
    
    <blockquote>
      <p>Collect a set of trajectories by executing the current policy</p>
      
      <p>At each timestep in each trajectory, compute</p>
      
      <blockquote>
        <p>the return <span class="math-container">$R_{t}= \sum_{t'=t}^{T-1}\gamma^{t'-t}r_{t'}$</span></p>
        
        <p>the advantage estimate <span class="math-container">$\hat{A}_{t} = R_{t} - b(s_{t})$</span></p>
      </blockquote>
      
      <p>Re-fit the baseline, by minimizing <span class="math-container">$\lVert b(s_{t}) - R_{t} \rVert^{2}$</span></p>
      
      <blockquote>
        <p>summed over all trajectories and timesteps.</p>
      </blockquote>
      
      <p>Update the policy, using a policy gradient estimate <span class="math-container">$\hat{g}$</span>,</p>
      
      <blockquote>
        <p>which is a sum of terms <span class="math-container">$\nabla_{\theta}log\pi(a_{t}|s_{t},\theta)\hat{A_{t}}$</span></p>
      </blockquote>
    </blockquote>
  </blockquote>
  
  <ul>
  <li>At line 6, advantage estimate is computed by subtracting baseline from the returns</li>
  <li>At line 7, baseline is re-fit minimizing mean squared error between state dependent baseline and return</li>
  <li>At line 8, we update the policy using advantage estimate from line <strong>6</strong></li>
  </ul>
  
  <p>So is the baseline expected to be used in the next iteration when our policy has <em>changed</em>?</p>
  
  <p>To compute the advantage we subtract the state value <span class="math-container">$V(s_{t})$</span> from the action value <span class="math-container">$Q(s_{t},a_{t})$</span>, under the <em>same</em> policy, then why is the old baseline used here in advantage estimation?</p>
  
Tags: <reinforcement-learning><policy-gradients>
Title: In vanilla policy gradient is the baseline lagging behind the policy?
UserDisplayName: junior-flight
ThreadText: # In vanilla policy gradient is the baseline lagging behind the policy?
  
  (Asked by: username_0 on 2020-05-22)
  
  Vanilla policy gradient algorithm (using baseline to reduce variance) acc to [here](http://joschu.net/docs/thesis.pdf) (page 16)
  
  > Initialize policy parameter θ, baseline b
  >
  > for iteration=1, 2, . . . do
  >
  > > Collect a set of trajectories by executing the current policy
  > >
  > > At each timestep in each trajectory, compute
  > >
  > > > the return $R\_{t}= \sum\_{t'=t}^{T-1}\gamma^{t'-t}r\_{t'}$
  > > >
  > > > the advantage estimate $\hat{A}\_{t} = R\_{t} - b(s\_{t})$
  > >
  > > Re-fit the baseline, by minimizing $\lVert b(s\_{t}) - R\_{t} \rVert^{2}$
  > >
  > > > summed over all trajectories and timesteps.
  > >
  > > Update the policy, using a policy gradient estimate $\hat{g}$,
  > >
  > > > which is a sum of terms $\nabla\_{\theta}log\pi(a\_{t}|s\_{t},\theta)\hat{A\_{t}}$
  
  * At line 6, advantage estimate is computed by subtracting baseline from the returns
  * At line 7, baseline is re-fit minimizing mean squared error between state dependent baseline and return
  * At line 8, we update the policy using advantage estimate from line **6**
  
  So is the baseline expected to be used in the next iteration when our policy has *changed*?
  
  To compute the advantage we subtract the state value $V(s\_{t})$ from the action value $Q(s\_{t},a\_{t})$, under the *same* policy, then why is the old baseline used here in advantage estimation?
  
  **Tags:** reinforcement-learning, policy-gradients
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-05-22. Score: 0 | Accepted answer
  
  > So is the baseline expected to be used in the next iteration when our policy has *changed*?
  
  Yes.
  
  > To compute the advantage we subtract the state value $V(s\_{t})$ from the action value $Q(s\_{t},a\_{t})$, under the *same* policy, then why is the old baseline used here in advantage estimation?
  
  The precise value of the baseline is not that important. What is important is that the baseline does not depend on the action choice, $a$, so it does not impact the gradient estimations or update steps for the policy function you are trying to improve.
  
  You could in theory use a fixed offset instead of $V(s)$, or any arbitrary function that does not depend on $a$. In some settings the average reward $\bar{R}$ seen so far is used.
  
  Using a rough approximation to $V(s)$ - and thus an approximate advantage function overall is useful as it removes a large source of variance in gradient estimates (the inherent value of the current state under the current policy, which is irrelevant to search for *adjustments* to that policy). The more accurate $V(s)$ is, the lower variance, thus faster and more reliable convergence, so you do want it to be a good estimate. But a little bit of lag behind policy updates is acceptable and does not break the algorithm.
  
  For more on this, see [Sutton & Barto, chapter 13, section 13.4](http://incompleteideas.net/book/RLbook2020.pdf).
  
  
  ---
  
  ### Answer by username_2 on 2023-08-26. Score: 1
  
  To be honest I was also struggling with this question for quite some time and I think that it was written this way mostly because of historic reasons. Once you have the advantage $A\_t = \sum\_{i=t}^{T} r\_{i+1} - V\_\phi(s)$, you can use it directly to update both the value and the policy networks:
  
  \begin{align}
  &w = w + \alpha A\_t \nabla V\_\phi(s) \\
  &\theta = \theta + \alpha A\_t \nabla \log \pi\_\theta(a|s)
  \end{align}
  
  To see why this is true note that you update the value network using the squared-error loss $L^{VF} = 0.5(\sum r\_{t+1} - V\_\phi(s))^2$, and when you differentiate this you get simply $A\_t$. And the loss for the policy network is simply $L^{PG} = \log \pi\_\theta(a|s) A\_t$.
  
  Essentially you evaluate the value network only one time, instead of two times, which probably back at the time when this algorithm was invented (e.g. in the 1990s) was a big deal.
  
  If I were to implement vpg with baseline, I would first update the value network and only then calculate the advantage to update the policy network, i.e. I would reshuffle the steps so that I use the updated baseline. However, I would never use vpg with baseline. I don't think anyone has ever used it for something practical and I don't know of any papers that show SotA results with it.
  
  Note that if you have the value network, instead of using it merely as a baseline, you would be much better off using it to bootstrap the estimate of the return, i.e. $A\_t = r\_{t+1} + V(s\_{t+1}) - V(s\_t).$  
  If you take a look at the pseudocode for this algorithm you can see that, again, people update the value network after they compute the advantage. Here, you could again update the value network before that, but now you have to be extra careful. When computing the advantage you have to use the old value network for bootstrapping and the new value network only for the baseline: $A\_t = r\_{t+1} + V\_{old}(s\_{t+1}) - V\_{new}(s\_t).$ A very brief explanation for this is given at the end of page 55 in John Schulman's phd thesis [that you shared](http://joschu.net/docs/thesis.pdf).
  So in this case, I would just go on and use the old network and not overthink it.
  
  Even better than a one-step bootstrap, you could do an n-step bootstrap: $A\_t^{(n)} = r\_{t+1} + r\_{t+2} + \cdots + r\_{t+n} + V(s\_{t+n}) - V(s\_t)$ and then use a generalized advantage estimation $A^{GAE} = \sum\_{n=1}^{\inf} (1-\lambda) \lambda ^ {n-1} A\_{t}^{(n)}$. When computing $A^{GAE}$ the value network is used both for bootstrapping and as a baseline and it is not possible to distinguish between the two uses because they are cancelled out in a telescoping sum. This means that the only correct way would be to update the value network after you compute the advantage.
  
  So you can see that once you have the value network you get from a poor algorithm to state-of-the-art with one simple equation. See [here](https://pi-tau.github.io/posts/actor-critic/) if you want to read more.
  
  
  ---
  
  