Answers.Body: <p>When the authors write go from
  <span class="math-container">$$\nabla_{\theta}J \propto \sum_s \mu(s) \sum_a q_{\pi}(s,a)\nabla_{\theta}\pi(a|s;\theta)\;$$</span>
  to
  <span class="math-container">$$\nabla_{\theta}J = E_{\pi}\left[\sum_a q_{\pi}(S_t,a) \nabla_{\theta}\pi(a|S_t;\theta)\right]\;$$</span>
  they are simply taking an expectation where the only random variable is the state <span class="math-container">$S_t$</span>. This is because, as they say in the book, the Policy Gradient Theorem is a sum over a state distribution, therefore it can be written as an expectation. The expectation is over the states wrt the state distribution; the <span class="math-container">$\pi$</span> subscript here does not mean we are taking expectation with respect to the action policy <span class="math-container">$\pi$</span>, they use the notation <span class="math-container">$\pi$</span> to emphasise that the state distribution is induced by the current policy <span class="math-container">$\pi$</span> -- this is what makes REINFORCE an on-policy algorithm because the state distribution with which we take the expectation must come from the current policy <span class="math-container">$\pi$</span>, not some arbitrary distribution.</p>
  <p>Now, as you don't seem to understand how they go from this to the Stochastic Gradient <em>Ascent</em> (SGA) (NOT descent as you keep referring to it as) update rule in the book, I will explain further. As you rightly say we can go from my previous equation to the following
  <span class="math-container">$$\nabla_{\theta}J = E_{\pi}\left[G_t \nabla_{\theta}\log\pi(A_t|S_t;\theta)\right]\;$$</span>
  by doing some re-arranging and noting that <span class="math-container">$G_t$</span> is an unbiased estimate of <span class="math-container">$q_\pi(s, a)$</span> when taking expectation over the actions and states. Now, this expectation we have is with respect to the actions and states; your first mistake is that you have made it an expectation over the actions without signifying that the action is now a random variable (maybe you could look up expectations and random variables to understand really what this means).</p>
  <p>Your second mistake is by not seeing how you can use this gradient to optimise the policy parameters. As we want to <em>maximise</em> this objective we want to perform SGA to it; that is we want to perform
  <span class="math-container">$$\theta = \theta + \alpha \nabla_\theta J\;;$$</span>
  <em>but</em> <span class="math-container">$\nabla_\theta J$</span> is an expectation. It would be costly to evaluate this at every single state-action pair that we see in an episode, thus we only perform SGA for one state-action tuple at a time. Now, you might think that this is not correct, but it is in fact an unbiased estimate as we will sample lots of state-action pairs; this kind of estimation is known as <a href="https://en.wikipedia.org/wiki/Monte_Carlo_method" rel="nofollow noreferrer">Monte Carlo Sampling</a>. Therefore, we can write our REINFORCE update rule as</p>
  <p><span class="math-container">$$\theta = \theta + \alpha G_t \nabla_\theta \log \pi(A_t | S_t; \theta)\;;$$</span></p>
  <p>which is exactly the update rule in the book.</p>
  <p>Note that at the start of the chapter in Sutton and Barto they also make it clear that we wish to perform
  <span class="math-container">$$\theta = \theta + \alpha \hat{\nabla J(\theta)} \;;$$</span>
  where <span class="math-container">$\hat{\nabla J(\theta)}$</span> is a stochastic estimate whose expectation approximates the true gradient; this is exactly what we are doing, <span class="math-container">$G_t \nabla_\theta \log \pi(A_t | S_t; \theta)$</span> is our stochastic estimate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-31T12:13:10.977Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/122498/discussion-on-answer-by-david-ireland-how-to-simplify-policy-gradient-theoram-to).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-30T18:10:36.047Z
Answers.LastActivityDate: 2021-03-30T23:40:17.600Z
Answers.LastEditDate: 2021-03-30T23:40:17.600Z
Answers.UserDisplayName: David
Body: <p>In &quot;Introduction to Reinforcement Learning&quot; (Richard Sutton) section 13.3(Reinforce algorithm) they have the following equation:</p>
  <p><span class="math-container">\begin{align}
   \nabla_{\theta}J &amp;\propto \sum_s \mu(s) \sum_a q_{\pi}(s,a)\nabla_{\theta}\pi(a|s,\theta) \\
                    &amp;= E_{\pi}[\sum_a q_{\pi}(S_t,a) \nabla_{\theta}\pi(a|S_t,\theta)] \tag{1}\label{1}
  \end{align}</span>
  But in my opinion equation 1 should be expectation over state distribution: <span class="math-container">$$E_{\mu}[\sum_a q_{\pi}(S_t,a) \nabla_{\theta}\pi(a|S_t,\theta)]$$</span>
  If I am right here then the rest of the lines follows like this:
  <span class="math-container">\begin{align}
  \nabla_{\theta}J &amp;= E_{\mu}[\sum_a q_{\pi}(S_t,a) \nabla_{\theta}\pi(a|S_t,\theta)] \\
                    &amp;= E_{\mu}[\sum_a \pi(a|S_t,\theta) q_{\pi}(S_t,a) \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}] \\
                     &amp;= E_{\mu}[E_{\pi}[q_{\pi}(S_t,A_t)\frac{\nabla_{\theta}\pi(A_t|S_t,\theta)}{\pi(A_t|s,\theta)}]]\\
                     &amp;= E_{\mu}[E_{\pi}[G_t\frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]]
  \end{align}</span>
  Now the final update rule using stochastic gradient descent will be:
  <span class="math-container">$$\triangle \theta = \alpha E_{\pi}[G_t\frac{\nabla_{\theta}\pi(A_t|S_t,\theta)}{\pi(A_t|S_t,\theta)}] \tag{2}$$</span>
  I think I am doing something wrong here because this equation 2 does not match with the book also with other materials. Can anyone please show me where I am doing wrong?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-30T16:53:38.070Z
Comments.Text: It's explained in the book literally above the equation: "Notice that the right-hand
  side of the policy gradient theorem is a sum over states weighted by how often the states occur under the target policy $\pi$, if $\pi$ is followed, then states will be encountered in these proportions".
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-30T17:45:28.713Z
Comments.Text: Can you please explain this to me ?
Comments.UserDisplayName: Swakshar Deb
Tags: <reinforcement-learning><deep-rl><policy-gradients><reinforce><sutton-barto>
Title: How to simplify policy gradient theorem to $E_{\pi}[G_t \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]$?
UserDisplayName: Swakshar Deb
ThreadText: # How to simplify policy gradient theorem to $E_{\pi}[G_t \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]$?
  
  (Asked by: username_0 on 2021-03-30)
  
  In "Introduction to Reinforcement Learning" (Richard Sutton) section 13.3(Reinforce algorithm) they have the following equation:
  
  \begin{align}
  \nabla\_{\theta}J &\propto \sum\_s \mu(s) \sum\_a q\_{\pi}(s,a)\nabla\_{\theta}\pi(a|s,\theta) \\
  &= E\_{\pi}[\sum\_a q\_{\pi}(S\_t,a) \nabla\_{\theta}\pi(a|S\_t,\theta)] \tag{1}\label{1}
  \end{align}
  But in my opinion equation 1 should be expectation over state distribution: $$E\_{\mu}[\sum\_a q\_{\pi}(S\_t,a) \nabla\_{\theta}\pi(a|S\_t,\theta)]$$
  If I am right here then the rest of the lines follows like this:
  \begin{align}
  \nabla\_{\theta}J &= E\_{\mu}[\sum\_a q\_{\pi}(S\_t,a) \nabla\_{\theta}\pi(a|S\_t,\theta)] \\
  &= E\_{\mu}[\sum\_a \pi(a|S\_t,\theta) q\_{\pi}(S\_t,a) \frac{\nabla\_{\theta}\pi(a|S\_t,\theta)}{\pi(a|S\_t,\theta)}] \\
  &= E\_{\mu}[E\_{\pi}[q\_{\pi}(S\_t,A\_t)\frac{\nabla\_{\theta}\pi(A\_t|S\_t,\theta)}{\pi(A\_t|s,\theta)}]]\\
  &= E\_{\mu}[E\_{\pi}[G\_t\frac{\nabla\_{\theta}\pi(a|S\_t,\theta)}{\pi(a|S\_t,\theta)}]]
  \end{align}
  Now the final update rule using stochastic gradient descent will be:
  $$\triangle \theta = \alpha E\_{\pi}[G\_t\frac{\nabla\_{\theta}\pi(A\_t|S\_t,\theta)}{\pi(A\_t|S\_t,\theta)}] \tag{2}$$
  I think I am doing something wrong here because this equation 2 does not match with the book also with other materials. Can anyone please show me where I am doing wrong?
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients, reinforce, sutton-barto
  
  > **Comments:**
  > **username_1**: It's explained in the book literally above the equation: "Notice that the right-hand side of the policy gradient theorem is a sum over states weighted by how often the states occur under the target policy $\pi$, if $\pi$ is followed, then states will be encountered in these proportions".  
  > **username_0**: Can you please explain this to me ?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-03-30. Score: 1
  
  When the authors write go from
  $$\nabla\_{\theta}J \propto \sum\_s \mu(s) \sum\_a q\_{\pi}(s,a)\nabla\_{\theta}\pi(a|s;\theta)\;$$
  to
  $$\nabla\_{\theta}J = E\_{\pi}\left[\sum\_a q\_{\pi}(S\_t,a) \nabla\_{\theta}\pi(a|S\_t;\theta)\right]\;$$
  they are simply taking an expectation where the only random variable is the state $S\_t$. This is because, as they say in the book, the Policy Gradient Theorem is a sum over a state distribution, therefore it can be written as an expectation. The expectation is over the states wrt the state distribution; the $\pi$ subscript here does not mean we are taking expectation with respect to the action policy $\pi$, they use the notation $\pi$ to emphasise that the state distribution is induced by the current policy $\pi$ -- this is what makes REINFORCE an on-policy algorithm because the state distribution with which we take the expectation must come from the current policy $\pi$, not some arbitrary distribution.
  
  Now, as you don't seem to understand how they go from this to the Stochastic Gradient *Ascent* (SGA) (NOT descent as you keep referring to it as) update rule in the book, I will explain further. As you rightly say we can go from my previous equation to the following
  $$\nabla\_{\theta}J = E\_{\pi}\left[G\_t \nabla\_{\theta}\log\pi(A\_t|S\_t;\theta)\right]\;$$
  by doing some re-arranging and noting that $G\_t$ is an unbiased estimate of $q\_\pi(s, a)$ when taking expectation over the actions and states. Now, this expectation we have is with respect to the actions and states; your first mistake is that you have made it an expectation over the actions without signifying that the action is now a random variable (maybe you could look up expectations and random variables to understand really what this means).
  
  Your second mistake is by not seeing how you can use this gradient to optimise the policy parameters. As we want to *maximise* this objective we want to perform SGA to it; that is we want to perform
  $$\theta = \theta + \alpha \nabla\_\theta J\;;$$
  *but* $\nabla\_\theta J$ is an expectation. It would be costly to evaluate this at every single state-action pair that we see in an episode, thus we only perform SGA for one state-action tuple at a time. Now, you might think that this is not correct, but it is in fact an unbiased estimate as we will sample lots of state-action pairs; this kind of estimation is known as [Monte Carlo Sampling](https://en.wikipedia.org/wiki/Monte_Carlo_method). Therefore, we can write our REINFORCE update rule as
  
  $$\theta = \theta + \alpha G\_t \nabla\_\theta \log \pi(A\_t | S\_t; \theta)\;;$$
  
  which is exactly the update rule in the book.
  
  Note that at the start of the chapter in Sutton and Barto they also make it clear that we wish to perform
  $$\theta = \theta + \alpha \hat{\nabla J(\theta)} \;;$$
  where $\hat{\nabla J(\theta)}$ is a stochastic estimate whose expectation approximates the true gradient; this is exactly what we are doing, $G\_t \nabla\_\theta \log \pi(A\_t | S\_t; \theta)$ is our stochastic estimate.
  
  > **Comments:**
  > **username_3**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/122498/discussion-on-answer-by-username_2-ireland-how-to-simplify-policy-gradient-theoram-to).  
  
  ---
  
  