Body: <p>In policy gradient theory subtracting state dependent baseline from Q(s,a) does not affect gradient of the objective function. I understand the proof shown below. One things is, if it's not affecting the gradient, then how this approach is better?
  <a href="https://i.sstatic.net/N0XUY.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/N0XUY.png" alt="Proof that state dependent baseline does not affect gradient of the objective funtion" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-20T14:11:23.967Z
Comments.Text: Short answer: variance reduction. Some details may be found, e.g., [here](https://danieltakeshi.github.io/2017/03/28/going-deeper-into-reinforcement-learning-fundamentals-of-policy-gradients/).
Comments.UserDisplayName: mikkola
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-20T15:48:13.200Z
Comments.Text: @mikkola thank you for your time end effort sir. I will check this out.
Comments.UserDisplayName: Davaasuren Nyamdavaa
Tags: <reinforcement-learning><policy-gradients>
Title: In the policy gradient method, state dependent baseline does not affect gradient of the objective function. Then how this is better approach?
UserDisplayName: Davaasuren Nyamdavaa
ThreadText: # In the policy gradient method, state dependent baseline does not affect gradient of the objective function. Then how this is better approach?
  
  (Asked by: username_0 on 2022-09-20)
  
  In policy gradient theory subtracting state dependent baseline from Q(s,a) does not affect gradient of the objective function. I understand the proof shown below. One things is, if it's not affecting the gradient, then how this approach is better?
  [![Proof that state dependent baseline does not affect gradient of the objective funtion](https://i.sstatic.net/N0XUY.png)](https://i.sstatic.net/N0XUY.png)
  
  **Tags:** reinforcement-learning, policy-gradients
  
  > **Comments:**
  > **username_1**: Short answer: variance reduction. Some details may be found, e.g., [here](https://danieltakeshi.github.io/2017/03/28/going-deeper-into-reinforcement-learning-fundamentals-of-policy-gradients/).  
  > **username_0**: @username_1 thank you for your time end effort sir. I will check this out.  
  
  ---
  
  ## 0 answers
  
  