Answers.Body: <p>The policy gradient states that
  <span class="math-container">$$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a q_\pi(s, a) \nabla\pi(a | s; \theta)\;$$</span>
  where the derivatives are taken wrt the parameter <span class="math-container">$\theta$</span>.</p>
  <p>Now, if we say incorporate a baseline we get
  <span class="math-container">$$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a \left( q_\pi(s, a) - b(s) \right)\nabla\pi(a | s; \theta)\;$$</span>
  and this does not effect the gradient at all. To see this, note that
  <span class="math-container">$$\sum_a b(s) \nabla\pi(a|s; \theta) = b(s) \nabla \sum_a \pi(a|s; \theta) = b(s) \nabla 1 = 0\;;$$</span>
  where all I have done is expand the bracketed terms inside the sum over <span class="math-container">$a$</span> from the second equation, and shown that the new term is equal to 0 -- thus the gradient is unchanged.</p>
  <p>If you really want to confirm this then you can fully write down the expansion of the second equation and use the trick I have shown you in my third equation to see that expanded second equation is equal to the first equation.</p>
  <p>I imagine that the EGLP lemma that the authors referred to will use a similar trick of a derivative of a probability distribution equalling to 0 when summing(/integrating) over the support of the random variable, which is what I have used here to go from <span class="math-container">$\nabla \sum_a\pi(a|s; \theta) = \nabla 1$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-07T17:46:43.780Z
Answers.LastActivityDate: 2021-07-07T17:46:43.780Z
Answers.UserDisplayName: David
Body: <p>On <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#baselines-in-policy-gradients" rel="nofollow noreferrer">the OpenAI's Spinning Up</a>, they justify the fact that adding a baseline <span class="math-container">$b(s_t)$</span> in the policy gradient doesn't change its gradient by saying that this is</p>
  <blockquote>
  <p>an immediate consequence of the EGLP Lemma</p>
  </blockquote>
  <p>However, I did not manage to prove it with this lemma. Can somebody help me, please?</p>
  <p>The proof is trivial when <span class="math-container">$b$</span> is a constant, but I struggle to derive it whenever <span class="math-container">$b$</span> is a function of the current state <span class="math-container">$s$</span> because you can't take it out of the integral.</p>
  
Tags: <policy-gradients><proofs><open-ai><reinforce>
Title: Why adding a baseline doesn't affect the policy gradient?
UserDisplayName: Thomas Hustache
ThreadText: # Why adding a baseline doesn't affect the policy gradient?
  
  (Asked by: username_0 on 2021-07-07)
  
  On [the OpenAI's Spinning Up](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#baselines-in-policy-gradients), they justify the fact that adding a baseline $b(s\_t)$ in the policy gradient doesn't change its gradient by saying that this is
  
  > an immediate consequence of the EGLP Lemma
  
  However, I did not manage to prove it with this lemma. Can somebody help me, please?
  
  The proof is trivial when $b$ is a constant, but I struggle to derive it whenever $b$ is a function of the current state $s$ because you can't take it out of the integral.
  
  **Tags:** policy-gradients, proofs, open-ai, reinforce
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-07. Score: 2
  
  The policy gradient states that
  $$\nabla J(\theta) \propto \sum\_s \mu(s) \sum\_a q\_\pi(s, a) \nabla\pi(a | s; \theta)\;$$
  where the derivatives are taken wrt the parameter $\theta$.
  
  Now, if we say incorporate a baseline we get
  $$\nabla J(\theta) \propto \sum\_s \mu(s) \sum\_a \left( q\_\pi(s, a) - b(s) \right)\nabla\pi(a | s; \theta)\;$$
  and this does not effect the gradient at all. To see this, note that
  $$\sum\_a b(s) \nabla\pi(a|s; \theta) = b(s) \nabla \sum\_a \pi(a|s; \theta) = b(s) \nabla 1 = 0\;;$$
  where all I have done is expand the bracketed terms inside the sum over $a$ from the second equation, and shown that the new term is equal to 0 -- thus the gradient is unchanged.
  
  If you really want to confirm this then you can fully write down the expansion of the second equation and use the trick I have shown you in my third equation to see that expanded second equation is equal to the first equation.
  
  I imagine that the EGLP lemma that the authors referred to will use a similar trick of a derivative of a probability distribution equalling to 0 when summing(/integrating) over the support of the random variable, which is what I have used here to go from $\nabla \sum\_a\pi(a|s; \theta) = \nabla 1$.
  
  
  ---
  
  