Answers.Body: <p>Using the <a href="https://en.wikipedia.org/wiki/Law_of_total_expectation" rel="nofollow noreferrer">law of iterated expectations</a> one has:</p>
  
  <p><span class="math-container">$\triangledown _\theta \sum_{t=1}^T \mathbb{E}_{(s_t,a_t) \sim p(s_t,a_t)} [b(s_t)] = \nabla_\theta \sum_{t=1}^T \mathbb{E}_{s_t \sim p(s_t)} \left[ \mathbb{E}_{a_t \sim \pi_\theta(a_t | s_t)} 
   \left[ b(s_t) \right]\right] =$</span></p>
  
  <p>written with integrals and moving the gradient inside (linearity) you get</p>
  
  <p><span class="math-container">$= \sum_{t=1}^T \int_{s_t} p(s_t) \left(\int_{a_t} \nabla_\theta b(s_t) \pi_\theta(a_t | s_t) da_t \right)ds_t =$</span></p>
  
  <p>you can now move <span class="math-container">$\nabla_\theta$</span> (due to linearity) and <span class="math-container">$b(s_t)$</span> (does not depend on <span class="math-container">$a_t$</span>) form the inner integral to the outer one:</p>
  
  <p><span class="math-container">$= 
  \sum_{t=1}^T \int_{s_t} p(s_t) b(s_t) \nabla_\theta \left(\int_{a_t} \pi_\theta(a_t | s_t)  da_t \right)ds_t= $</span></p>
  
  <p><span class="math-container">$\pi_\theta(a_t | s_t)$</span> is a (conditional) probability density function, so integrating over all <span class="math-container">$a_t$</span> for a given fixed state <span class="math-container">$s_t$</span> equals <span class="math-container">$1$</span>:</p>
  
  <p><span class="math-container">$= 
  \sum_{t=1}^T \int_{s_t} p(s_t) b(s_t) \nabla_\theta 1 ds_t = $</span></p>
  
  <p>Now <span class="math-container">$\nabla_\theta1 = 0$</span>, which concludes the proof.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-22T09:32:22.770Z
Answers.LastActivityDate: 2020-06-10T16:42:11.700Z
Answers.LastEditDate: 2020-06-10T16:42:11.700Z
Answers.UserDisplayName: Andrei Poehlmann
Answers.Body: <p>It appears that the homework was due two days prior to this answer's writing, but in case it is still relevant in some way, the relevant class notes (which would have been useful if provided in the question along with the homework) <a href="http://rail.eecs.berkeley.edu/deeprlcourse-fa17/f17docs/lecture_3_rl_intro.pdf" rel="nofollow noreferrer">are here</a>.</p>
  
  <p>The first instance of expectation placed on the student is, "Please show equation 12 by using the law of iterated expectations, breaking <span class="math-container">$\mathbb{E}_{\tau \sim p \theta(\tau)}$</span> by decoupling the state-action marginal from the rest of the trajectory."  Equation 12 is this.</p>
  
  <p><span class="math-container">$\sum_{t = 1}^{T} E_{\tau \sim p \theta(\tau)} [\nabla_\theta \log \pi_\theta(a_t|s_t)(b(s_t))] = 0$</span></p>
  
  <p>The class notes identifies <span class="math-container">$\pi_\theta(a_t|s_t)$</span> as the state-action marginal.  It is not a proof sought, but a sequence of algebraic steps to perform the decoupling and show the degree to which independence of the state-action marginal can be achieved.</p>
  
  <p>This exercise is a preparation for the next step in the homework and draws only on the review of CS189, Burkeley's Introduction to Machine Learning course, which does not contain the Law of Total Expectation in its syllabus or class notes.</p>
  
  <p>All the relevant information is in the above link for class notes and requires only intermediate algebra.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-22T01:05:57.067Z
Answers.LastActivityDate: 2018-09-22T04:30:41.657Z
Answers.LastEditDate: 2018-09-22T04:30:41.657Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>In the <a href="http://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw2.pdf" rel="nofollow noreferrer">homework for the Berkeley RL class</a>, problem 1, it asks you to show that the policy gradient is still unbiased if the baseline subtracted is a function of the state at time step <span class="math-container">$t$</span>.</p>
  
  <p><span class="math-container">$$ \triangledown _\theta \sum_{t=1}^T \mathbb{E}_{(s_t,a_t) \sim p(s_t,a_t)} [b(s_t)] = 0   $$</span></p>
  
  <p>I am struggling through what the first step of such a proof might be. </p>
  
  <p>Can someone point me in the right direction? My initial thought was to somehow use the <a href="https://en.wikipedia.org/wiki/Law_of_total_expectation" rel="nofollow noreferrer">law of total expectation</a> to make the expectation of <span class="math-container">$b(s_t)$</span> conditional on <span class="math-container">$T$</span>, but I am not sure.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-10T09:51:57.100Z
Comments.Text: Don't really have much time to write down the exact equations and format it (maybe later if it is still not answered) with LaTeX but here is a hint. You want to have that the sum does not depend on the policy so that the derivative would be 0. So you somehow try to express things using the policy p(s,a). The answer btw can also be found in Sutton's RL Intro book in the policy gradient chapter.
Comments.UserDisplayName: Hai Nguyen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-10T10:51:18.360Z
Comments.Text: Thank you very much! I will use that hint to get started, as well as thank you for telling me about it being in Sutton RL. I am reading that book and it is quite excellent!
Comments.UserDisplayName: Laura C
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-07T12:14:21.923Z
Comments.Text: @LauraC could you please elaborate how you arrived at this formulation? Because in the homework (at least as of now), the original equation is different and I don't know how to get to your version. Thanks!
Comments.UserDisplayName: Max Semikin
Tags: <reinforcement-learning><policy-gradients><proofs>
Title: Why is baseline conditional on state at some timestep unbiased?
UserDisplayName: Laura C
ThreadText: # Why is baseline conditional on state at some timestep unbiased?
  
  (Asked by: username_0 on 2018-09-09)
  
  In the [homework for the Berkeley RL class](http://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw2.pdf), problem 1, it asks you to show that the policy gradient is still unbiased if the baseline subtracted is a function of the state at time step $t$.
  
  $$ \triangledown \_\theta \sum\_{t=1}^T \mathbb{E}\_{(s\_t,a\_t) \sim p(s\_t,a\_t)} [b(s\_t)] = 0 $$
  
  I am struggling through what the first step of such a proof might be.
  
  Can someone point me in the right direction? My initial thought was to somehow use the [law of total expectation](https://en.wikipedia.org/wiki/Law_of_total_expectation) to make the expectation of $b(s\_t)$ conditional on $T$, but I am not sure.
  
  **Tags:** reinforcement-learning, policy-gradients, proofs
  > **Comments:**
  > **username_1**: Don't really have much time to write down the exact equations and format it (maybe later if it is still not answered) with LaTeX but here is a hint. You want to have that the sum does not depend on the policy so that the derivative would be 0. So you somehow try to express things using the policy p(s,a). The answer btw can also be found in Sutton's RL Intro book in the policy gradient chapter.  
  > **username_0**: Thank you very much! I will use that hint to get started, as well as thank you for telling me about it being in Sutton RL. I am reading that book and it is quite excellent!  
  > **username_2**: @LauraC could you please elaborate how you arrived at this formulation? Because in the homework (at least as of now), the original equation is different and I don't know how to get to your version. Thanks!  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2018-09-22. Score: 7
  
  Using the [law of iterated expectations](https://en.wikipedia.org/wiki/Law_of_total_expectation) one has:
  
  $\triangledown \_\theta \sum\_{t=1}^T \mathbb{E}\_{(s\_t,a\_t) \sim p(s\_t,a\_t)} [b(s\_t)] = \nabla\_\theta \sum\_{t=1}^T \mathbb{E}\_{s\_t \sim p(s\_t)} \left[ \mathbb{E}\_{a\_t \sim \pi\_\theta(a\_t | s\_t)}
  \left[ b(s\_t) \right]\right] =$
  
  written with integrals and moving the gradient inside (linearity) you get
  
  $= \sum\_{t=1}^T \int\_{s\_t} p(s\_t) \left(\int\_{a\_t} \nabla\_\theta b(s\_t) \pi\_\theta(a\_t | s\_t) da\_t \right)ds\_t =$
  
  you can now move $\nabla\_\theta$ (due to linearity) and $b(s\_t)$ (does not depend on $a\_t$) form the inner integral to the outer one:
  
  $=
  \sum\_{t=1}^T \int\_{s\_t} p(s\_t) b(s\_t) \nabla\_\theta \left(\int\_{a\_t} \pi\_\theta(a\_t | s\_t) da\_t \right)ds\_t= $
  
  $\pi\_\theta(a\_t | s\_t)$ is a (conditional) probability density function, so integrating over all $a\_t$ for a given fixed state $s\_t$ equals $1$:
  
  $=
  \sum\_{t=1}^T \int\_{s\_t} p(s\_t) b(s\_t) \nabla\_\theta 1 ds\_t = $
  
  Now $\nabla\_\theta1 = 0$, which concludes the proof.
  
  
  ---
  
  ### Answer by username_4 on 2018-09-22. Score: 1
  
  It appears that the homework was due two days prior to this answer's writing, but in case it is still relevant in some way, the relevant class notes (which would have been useful if provided in the question along with the homework) [are here](http://rail.eecs.berkeley.edu/deeprlcourse-fa17/f17docs/lecture_3_rl_intro.pdf).
  
  The first instance of expectation placed on the student is, "Please show equation 12 by using the law of iterated expectations, breaking $\mathbb{E}\_{\tau \sim p \theta(\tau)}$ by decoupling the state-action marginal from the rest of the trajectory." Equation 12 is this.
  
  $\sum\_{t = 1}^{T} E\_{\tau \sim p \theta(\tau)} [\nabla\_\theta \log \pi\_\theta(a\_t|s\_t)(b(s\_t))] = 0$
  
  The class notes identifies $\pi\_\theta(a\_t|s\_t)$ as the state-action marginal. It is not a proof sought, but a sequence of algebraic steps to perform the decoupling and show the degree to which independence of the state-action marginal can be achieved.
  
  This exercise is a preparation for the next step in the homework and draws only on the review of CS189, Burkeley's Introduction to Machine Learning course, which does not contain the Law of Total Expectation in its syllabus or class notes.
  
  All the relevant information is in the above link for class notes and requires only intermediate algebra.
  
  
  ---
  
  