Answers.Body: <p>I went into the pytorch code for the spinning up implementation of vanilla policy gradient and from what I could understand, found that they use a learning rate of 1e-3 for training the baseline and do a gradient descent 80 times on the same dataset by default with no termination criteria. </p>
  
  <p>Also it is usually impossible to fit the value function completely as we are using a function approximator. The main point is not to get too carried away to get the lowest loss as the main improvement in the agent's performance will happen by doing policy gradient descents rather than trying to aggressively get the accurate value function for the baseline.</p>
  
  <p>Link for the implementation:
  <a href="https://github.com/openai/spinningup/blob/master/spinup/algos/pytorch/vpg/vpg.py" rel="nofollow noreferrer">spinning up implementation of vpg</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-18T07:29:12.990Z
Answers.LastActivityDate: 2020-05-18T07:29:12.990Z
Answers.UserDisplayName: junior-flight
Body: <p>How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods? </p>
  
  <p>I'm using this equation to compute the policy gradient:</p>
  
  <p><span class="math-container">$$
  \nabla_{\theta} J\left(\pi_{\theta}\right)=\underset{\tau \sim \pi_{\theta}}{\mathrm{E}}\left[\sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}\left(a_{t} | s_{t}\right)\left(\sum_{t^{\prime}=t}^{T} R\left(s_{t^{\prime}}, a_{t^{\prime}}, s_{t^{\prime}+1}\right)-b\left(s_{t}\right)\right)\right]
  $$</span></p>
  
  <p><a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#baselines-in-policy-gradients" rel="nofollow noreferrer">Here</a> is mentioned to use one or more gradient steps, so is it a hyper-parameter to be found using random search? </p>
  
  <p>Is there some way we can use an adaptive method to find out when to stop?</p>
  
  <p>In an experiment to train Cartpole-v2 using a policy gradient with baseline, I found the results are better when applying 5 updates than when only a single update was applied.</p>
  
  <p>Note: I am referring to the number of updates to take on a single batch of q values encountered across trajectories collected using current policy.</p>
  
Tags: <reinforcement-learning><gradient-descent><policy-gradients><reinforce><adam>
Title: How long should the state-dependent baseline for policy gradient methods be trained at each iteration?
UserDisplayName: junior-flight
ThreadText: # How long should the state-dependent baseline for policy gradient methods be trained at each iteration?
  
  (Asked by: username_0 on 2020-05-08)
  
  How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods?
  
  I'm using this equation to compute the policy gradient:
  
  $$
  \nabla\_{\theta} J\left(\pi\_{\theta}\right)=\underset{\tau \sim \pi\_{\theta}}{\mathrm{E}}\left[\sum\_{t=0}^{T} \nabla\_{\theta} \log \pi\_{\theta}\left(a\_{t} | s\_{t}\right)\left(\sum\_{t^{\prime}=t}^{T} R\left(s\_{t^{\prime}}, a\_{t^{\prime}}, s\_{t^{\prime}+1}\right)-b\left(s\_{t}\right)\right)\right]
  $$
  
  [Here](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#baselines-in-policy-gradients) is mentioned to use one or more gradient steps, so is it a hyper-parameter to be found using random search?
  
  Is there some way we can use an adaptive method to find out when to stop?
  
  In an experiment to train Cartpole-v2 using a policy gradient with baseline, I found the results are better when applying 5 updates than when only a single update was applied.
  
  Note: I am referring to the number of updates to take on a single batch of q values encountered across trajectories collected using current policy.
  
  **Tags:** reinforcement-learning, gradient-descent, policy-gradients, reinforce, adam
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-05-18. Score: 0
  
  I went into the pytorch code for the spinning up implementation of vanilla policy gradient and from what I could understand, found that they use a learning rate of 1e-3 for training the baseline and do a gradient descent 80 times on the same dataset by default with no termination criteria.
  
  Also it is usually impossible to fit the value function completely as we are using a function approximator. The main point is not to get too carried away to get the lowest loss as the main improvement in the agent's performance will happen by doing policy gradient descents rather than trying to aggressively get the accurate value function for the baseline.
  
  Link for the implementation:
  [spinning up implementation of vpg](https://github.com/openai/spinningup/blob/master/spinup/algos/pytorch/vpg/vpg.py)
  
  
  ---
  
  