Answers.Body: <p>PPG can update the value function in the auxiliary phase because it uses a technique called &quot;<strong>reparameterization</strong>&quot; which allows for the gradient of the value function to be computed more efficiently. Additionally, PPG can take advantage of the structure of the problem to provide a better estimate of the value function.</p>
  <p>I think you can train the value network on data from any policy, but
  it will have more variance. If you have access to more data, it could help to smooth out the value function.</p>
  <blockquote>
  <p>One approach would be to update the <strong>value network</strong> after each training phase of the policy network. This would ensure that the value network is always fitted to the current policy. However, this might not be practical if the training phases are very long.</p>
  </blockquote>
  <blockquote>
  <p>Another approach would be to fit the value network using a <strong>rolling window</strong> of data from the previous training phases. This would strike a balance between stability and accuracy.</p>
  </blockquote>
  <p><code>&quot;Wouldn't it hurt the stability of fitting value function?&quot;</code><span class="math-container">$\rightarrow$</span>
  <span class="math-container">$\color{red}{\textbf{No}}$</span>, it should not hurt the stability of fitting the value function. In fact, using data from different policies can actually help to improve the stability of the value function estimate. This is because of &quot;reparameterization&quot; which allows for the gradient of the value function to be computed more efficiently as discussed above.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-18T03:59:42.180Z
Answers.Text: Aren't we estimating the value function w.r.t $\pi$? If we fit the value function with returns coming from different policies, we are moving towards different $V^{\pi}$?
Answers.UserDisplayName: Magi Feeney
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-17T20:36:46.973Z
Answers.LastActivityDate: 2022-10-17T20:36:46.973Z
Answers.UserDisplayName: Faizy
Body: <p>My questions is that how could we train the value network (separated from shared network) by using data from previous policies, which varies a lot since we collect data from different policies with many training phases done to start a auxiliary phase, especially returns are calculated by those policies. Wouldn't it hurt the stability of fitting value function?</p>
  
Tags: <deep-rl><actor-critic-methods><proximal-policy-optimization>
Title: Why Phasic Policy Gradient (PPG) can update value function in auxiliary phase?
UserDisplayName: Magi Feeney
ThreadText: # Why Phasic Policy Gradient (PPG) can update value function in auxiliary phase?
  
  (Asked by: username_0 on 2022-09-15)
  
  My questions is that how could we train the value network (separated from shared network) by using data from previous policies, which varies a lot since we collect data from different policies with many training phases done to start a auxiliary phase, especially returns are calculated by those policies. Wouldn't it hurt the stability of fitting value function?
  
  **Tags:** deep-rl, actor-critic-methods, proximal-policy-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-17. Score: 0
  
  PPG can update the value function in the auxiliary phase because it uses a technique called "**reparameterization**" which allows for the gradient of the value function to be computed more efficiently. Additionally, PPG can take advantage of the structure of the problem to provide a better estimate of the value function.
  
  I think you can train the value network on data from any policy, but
  it will have more variance. If you have access to more data, it could help to smooth out the value function.
  
  > One approach would be to update the **value network** after each training phase of the policy network. This would ensure that the value network is always fitted to the current policy. However, this might not be practical if the training phases are very long.
  
  > Another approach would be to fit the value network using a **rolling window** of data from the previous training phases. This would strike a balance between stability and accuracy.
  
  `"Wouldn't it hurt the stability of fitting value function?"`$\rightarrow$
  $\color{red}{\textbf{No}}$, it should not hurt the stability of fitting the value function. In fact, using data from different policies can actually help to improve the stability of the value function estimate. This is because of "reparameterization" which allows for the gradient of the value function to be computed more efficiently as discussed above.
  
  > **Comments:**
  > **username_0**: Aren't we estimating the value function w.r.t $\pi$? If we fit the value function with returns coming from different policies, we are moving towards different $V^{\pi}$?  
  
  ---
  
  