Answers.Body: <p>The proof you are given in the above post is not wrong. It's just they skip some of the steps and directly written the final answer. Let me go through those steps:</p>
  <p>I will simplify some of the things to avoid complication but the generosity remains the same. Like I will think of the reward as only dependent on the current state, <span class="math-container">$s$</span>, and current action, <span class="math-container">$a$</span>. So, <span class="math-container">$r = r(s,a)$</span></p>
  <p>First, we will define the average reward as:
  <span class="math-container">$$r(\pi) = \sum_s \mu(s)\sum_a \pi(a|s)\sum_{s^{\prime}} P_{ss'}^{a} r $$</span>
  We can further simplify average reward as:
  <span class="math-container">$$r(\pi) = \sum_s \mu(s)\sum_a \pi(a|s)r(s,a) $$</span>
  My notation may be slightly different than the aforementioned slides since I'm only following Sutton's book on RL. Our objective function is:
  <span class="math-container">$$ J(\theta) = r(\pi) $$</span>
  We want to prove that:
  <span class="math-container">$$ \nabla_{\theta} J(\theta) = \nabla_{\theta}r(\pi) = \sum_s \mu(s) \sum_a \nabla_{\theta}\pi(a|s) Q(s,a)$$</span></p>
  <p>Now let's start the proof:
  <span class="math-container">$$\nabla_{\theta}V(s) = \nabla_{\theta} \sum_{a} \pi(a|s) Q(s,a)$$</span>
  <span class="math-container">$$\nabla_{\theta}V(s) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s)  + \pi(a|s) \nabla_{\theta}Q(s,a)]$$</span>
  <span class="math-container">$$\nabla_{\theta}V(s) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s)  + \pi(a|s) \nabla_{\theta}[R(s,a) - r(\pi) + \sum_{s^{\prime}}P_{ss^{\prime}}^{a}V(s^{\prime})]]$$</span>
  <span class="math-container">$$\nabla_{\theta}V(s) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s)  + \pi(a|s) [- \nabla_{\theta}r(\pi) + \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime})]]$$</span>
  <span class="math-container">$$\nabla_{\theta}V(s) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s)  + \pi(a|s) \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime})] - \nabla_{\theta}r(\pi)\sum_{a}\pi(a|s)$$</span>
  Now we will rearrange this:
  <span class="math-container">$$\nabla_{\theta}r(\pi) = \sum_{a} [Q(s,a) \nabla_{\theta} \pi(a|s)  + \pi(a|s) \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime})] - \nabla_{\theta}V(s)$$</span>
  Multiplying both sides by <span class="math-container">$\mu(s)$</span> and summing over <span class="math-container">$s$</span>:
  <span class="math-container">$$\nabla_{\theta}r(\pi) \sum_{s}\mu(s)= \sum_{s}\mu(s) \sum_{a} Q(s,a) \nabla_{\theta} \pi(a|s)  + \sum_{s}\mu(s) \sum_a \pi(a|s) \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime}) - \sum_{s}\mu(s) \nabla_{\theta}V(s)$$</span>
  <span class="math-container">$$\nabla_{\theta}r(\pi) = \sum_{s}\mu(s) \sum_{a} Q(s,a) \nabla_{\theta} \pi(a|s)  + \sum_{s^{\prime}}\mu(s^{\prime})\nabla_{\theta}V(s^{\prime}) - \sum_{s}\mu(s) \nabla_{\theta}V(s)$$</span>
  Now we are there:
  <span class="math-container">$$\nabla_{\theta}r(\pi) = \sum_{s}\mu(s) \sum_{a} Q(s,a) \nabla_{\theta} \pi(a|s)$$</span>
  This is the policy gradient theoram for average reward formulation (ref. <a href="https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf" rel="nofollow noreferrer">Policy gradient</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T06:13:41.047Z
Answers.Text: Do you mean $\pi (a|s)$ instead of $\pi (s|a)$? Correct me if I'm misinterpreting!
Answers.UserDisplayName: stoic-santiago
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T06:24:43.610Z
Answers.Text: Also a little confused with the notation - why is $Q(s,a) = R(s,a) - r(\pi) + \sum_{s^{\prime}}P_{ss^{\prime}}^{a}V(s^{\prime})$ ?
Answers.UserDisplayName: stoic-santiago
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T06:30:19.887Z
Answers.Text: @cogito_ai I followed Sutton's notation. So, denoted policy as $\pi(a|s)$. For your case, $\pi(a|s) = \pi(s|a)$. But I think this is a mistake in this slide, it should be $\pi(a|s)$.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T06:31:35.413Z
Answers.Text: I think you've denoted the policy by $\pi(s|a)$ in your answer in multiple places - or is that something else? The slide shows $a|s$
Answers.UserDisplayName: stoic-santiago
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T06:32:38.123Z
Answers.Text: For average reward setting $Q(s,a) = r(s,a) - r(\pi) + \sum P_{ss'}v(s')$. See Sutton's book chapter on policy funciton approximation section average reward.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T06:33:36.660Z
Answers.Text: Sorry, it should be $\pi(a|s)$. It was my mistake.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T06:35:14.483Z
Answers.Text: How is $\sum_{s^{\prime}}\mu(s^{\prime})\nabla_{\theta}V(s^{\prime}) = \sum_{s}\mu(s) \sum_a \pi(s,a) \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime})$? Also which edition of Sutton's book are you using?
Answers.UserDisplayName: stoic-santiago
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T06:37:01.877Z
Answers.Text: Because this is the product rule of probability. This is not in the Sutton's book. I give the reference, check it for more details.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T06:39:33.947Z
Answers.Text: If you open the summation you can see clearly this is exactly telling us what is the probability of the agent to go to the state $s'$ from the state $s$.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T06:41:53.990Z
Answers.Text: Looks good! Lastly, in places where you've written $\pi(a,s)$, did you mean $\pi(a|s)$ there too?
Answers.UserDisplayName: stoic-santiago
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T06:42:33.997Z
Answers.Text: Yes, I make a mistake, it should be $\pi(a|s)$.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T06:44:10.477Z
Answers.Text: Your formulation ends with $Q(s,a)$, while the slide has $R(s,a)$ - why the difference? I understand the rest! It'd be great if you could elaborate on this!
Answers.UserDisplayName: stoic-santiago
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T06:50:31.463Z
Answers.Text: I do not follow this lecture for RL, So, I can not tell it now, first I should watch Deep RL Bootcamp. But I hope, if you think about it you may find the reason.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T07:11:03.917Z
Answers.Text: Okay, I watched their lecture. At the time 40.55 they, replace $\sum r(s,a)$ with $Q(s,a)$.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T08:58:11.243Z
Answers.Text: $Q(s,a)$ is an unbiased estimate of $R(s,a)$ — ie they are the same in expectation. @cogito_ai
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T16:36:40.527Z
Answers.LastActivityDate: 2020-09-04T00:11:03.963Z
Answers.LastEditDate: 2020-09-04T00:11:03.963Z
Answers.UserDisplayName: Swakshar Deb
Answers.Body: <p>The reason you are confused is because this is not the full derivation of the Policy Gradient Theorem. You are correct in thinking that <span class="math-container">$\mu(s)$</span> depends on the policy <span class="math-container">$\pi$</span> which in turn depends on the policy parameters <span class="math-container">$\theta$</span>, and so there <em>should</em> be a derivative of <span class="math-container">$\mu$</span> wrt <span class="math-container">$\theta$</span>, however the Policy Gradient Theorem doesn't require you to take this derivative.</p>
  <p>In fact, the great thing about the Policy Gradient Theorem is that the final result <em>does not</em> require you to take a derivative of the state distribution with respect to the policy parameters. I would encourage you to read and go through the derivation of the Policy Gradient Theorem from e.g. Sutton and Barto to see why you don't need to take the derivative.</p>
  <p><a href="https://i.sstatic.net/5IN1C.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5IN1C.png" alt="enter image description here" /></a></p>
  <p>Above is an image of the Policy Gradient Theorem proof from the Sutton and Barto book. If you carefully go through this line by line you will see that you are not required to take a derivative of the state distribution anywhere in the proof.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T03:22:55.830Z
Answers.Text: Could you please elaborate and possibly provide mathematical details? I've gone through the derivation in Sutton and Barto and I'm still unclear why the derivation above works.
Answers.UserDisplayName: stoic-santiago
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T08:53:37.853Z
Answers.Text: There’s not much to elaborate on. If you followed the Sutton and Barto proof you would see that you don’t need to take a derivative of the state distribution. I can’t really explain any clearer than just writing out the full proof which I am not going to do since that is not the point of this site. If you have a particular question about the proof that you find unclear then you should post that and I’d be happy to answer it.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T09:21:41.650Z
Answers.Text: Also, the above isn't a derivation. It is an extremely poor example, especially if they have not previously gone through the policy gradient theorem proof. What they have written (first two lines) gets the right answer but goes about it the wrong way.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T13:37:44.950Z
Answers.Text: I **understand** the proof written in Sutton and Barto's text. My question was - **why does the wrong proof (in my post) work?** I apologize if you've already explained this, but I still don't see why the wrong proof works. Sutton and Barto's proof doesn't require us to take the derivative of the state distribution anywhere, but *that is no reason for me to overlook the same* in the proof I've attached in my post. Please explain. Thanks!
Answers.UserDisplayName: stoic-santiago
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T13:40:21.410Z
Answers.Text: What you have posted _is not a proof_. They have _started_ with the final result of the Policy Gradient Theorem, and then they have derived the update that is used in REINFORCE. As I say, the source you are using does not look like something I would recommend. The equality on the first line should be replaced directly with the equality on the second line to avoid confusion. Regardless, if you want good source material for Reinforcement Learning then I would recommend you read Sutton and Barto to prepare you for more advanced topics you will find in papers.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-28T13:44:12.377Z
Answers.Text: @cogito_ai to answer your edit, what you have posted _is not a proof of the policy gradient theorem_. The author has _started_ with the final result in the policy gradient theorem, and has _derived_ something entirely separate.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-27T13:12:03.463Z
Answers.LastActivityDate: 2020-08-28T09:24:18.113Z
Answers.LastEditDate: 2020-08-28T09:24:18.113Z
Answers.UserDisplayName: David
Body: <p>I came across the following proof of what's commonly referred to as the <strong>log-derivative trick</strong> in policy-gradient algorithms, and I have a question -</p>
  <p><a href="https://i.sstatic.net/tqATW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tqATW.png" alt="enter image description here" /></a></p>
  <p>While transitioning from the first line to the second, the gradient with respect to policy parameters <span class="math-container">$\theta$</span> was pushed into the summation. What bothers me is how it skipped over <span class="math-container">$\mu (s)$</span>, the distribution of states - which (the way I understand it), <strong>is induced by the policy <span class="math-container">$\pi_\theta$</span> itself!</strong> Why then does it not depend on <span class="math-container">$\theta$</span>?</p>
  <p>Let me know what's going wrong! Thank you!</p>
  
Tags: <reinforcement-learning><policy-gradients><proofs>
Title: Why does (not) the distribution of states depend on the policy parameters that induce it?
UserDisplayName: stoic-santiago
ThreadText: # Why does (not) the distribution of states depend on the policy parameters that induce it?
  
  (Asked by: username_0 on 2020-08-27)
  
  I came across the following proof of what's commonly referred to as the **log-derivative trick** in policy-gradient algorithms, and I have a question -
  
  [![enter image description here](https://i.sstatic.net/tqATW.png)](https://i.sstatic.net/tqATW.png)
  
  While transitioning from the first line to the second, the gradient with respect to policy parameters $\theta$ was pushed into the summation. What bothers me is how it skipped over $\mu (s)$, the distribution of states - which (the way I understand it), **is induced by the policy $\pi\_\theta$ itself!** Why then does it not depend on $\theta$?
  
  Let me know what's going wrong! Thank you!
  
  **Tags:** reinforcement-learning, policy-gradients, proofs
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-08-28. Score: 1 | Accepted answer
  
  The proof you are given in the above post is not wrong. It's just they skip some of the steps and directly written the final answer. Let me go through those steps:
  
  I will simplify some of the things to avoid complication but the generosity remains the same. Like I will think of the reward as only dependent on the current state, $s$, and current action, $a$. So, $r = r(s,a)$
  
  First, we will define the average reward as:
  $$r(\pi) = \sum\_s \mu(s)\sum\_a \pi(a|s)\sum\_{s^{\prime}} P\_{ss'}^{a} r $$
  We can further simplify average reward as:
  $$r(\pi) = \sum\_s \mu(s)\sum\_a \pi(a|s)r(s,a) $$
  My notation may be slightly different than the aforementioned slides since I'm only following Sutton's book on RL. Our objective function is:
  $$ J(\theta) = r(\pi) $$
  We want to prove that:
  $$ \nabla\_{\theta} J(\theta) = \nabla\_{\theta}r(\pi) = \sum\_s \mu(s) \sum\_a \nabla\_{\theta}\pi(a|s) Q(s,a)$$
  
  Now let's start the proof:
  $$\nabla\_{\theta}V(s) = \nabla\_{\theta} \sum\_{a} \pi(a|s) Q(s,a)$$
  $$\nabla\_{\theta}V(s) = \sum\_{a} [Q(s,a) \nabla\_{\theta} \pi(a|s) + \pi(a|s) \nabla\_{\theta}Q(s,a)]$$
  $$\nabla\_{\theta}V(s) = \sum\_{a} [Q(s,a) \nabla\_{\theta} \pi(a|s) + \pi(a|s) \nabla\_{\theta}[R(s,a) - r(\pi) + \sum\_{s^{\prime}}P\_{ss^{\prime}}^{a}V(s^{\prime})]]$$
  $$\nabla\_{\theta}V(s) = \sum\_{a} [Q(s,a) \nabla\_{\theta} \pi(a|s) + \pi(a|s) [- \nabla\_{\theta}r(\pi) + \sum\_{s^{\prime}}P\_{ss^{\prime}}^{a}\nabla\_{\theta}V(s^{\prime})]]$$
  $$\nabla\_{\theta}V(s) = \sum\_{a} [Q(s,a) \nabla\_{\theta} \pi(a|s) + \pi(a|s) \sum\_{s^{\prime}}P\_{ss^{\prime}}^{a}\nabla\_{\theta}V(s^{\prime})] - \nabla\_{\theta}r(\pi)\sum\_{a}\pi(a|s)$$
  Now we will rearrange this:
  $$\nabla\_{\theta}r(\pi) = \sum\_{a} [Q(s,a) \nabla\_{\theta} \pi(a|s) + \pi(a|s) \sum\_{s^{\prime}}P\_{ss^{\prime}}^{a}\nabla\_{\theta}V(s^{\prime})] - \nabla\_{\theta}V(s)$$
  Multiplying both sides by $\mu(s)$ and summing over $s$:
  $$\nabla\_{\theta}r(\pi) \sum\_{s}\mu(s)= \sum\_{s}\mu(s) \sum\_{a} Q(s,a) \nabla\_{\theta} \pi(a|s) + \sum\_{s}\mu(s) \sum\_a \pi(a|s) \sum\_{s^{\prime}}P\_{ss^{\prime}}^{a}\nabla\_{\theta}V(s^{\prime}) - \sum\_{s}\mu(s) \nabla\_{\theta}V(s)$$
  $$\nabla\_{\theta}r(\pi) = \sum\_{s}\mu(s) \sum\_{a} Q(s,a) \nabla\_{\theta} \pi(a|s) + \sum\_{s^{\prime}}\mu(s^{\prime})\nabla\_{\theta}V(s^{\prime}) - \sum\_{s}\mu(s) \nabla\_{\theta}V(s)$$
  Now we are there:
  $$\nabla\_{\theta}r(\pi) = \sum\_{s}\mu(s) \sum\_{a} Q(s,a) \nabla\_{\theta} \pi(a|s)$$
  This is the policy gradient theoram for average reward formulation (ref. [Policy gradient](https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf)).
  
  > **Comments:**
  > **username_0**: Do you mean $\pi (a|s)$ instead of $\pi (s|a)$? Correct me if I'm misinterpreting!  
  > **username_0**: Also a little confused with the notation - why is $Q(s,a) = R(s,a) - r(\pi) + \sum_{s^{\prime}}P_{ss^{\prime}}^{a}V(s^{\prime})$ ?  
  > **username_1**: @cogito_ai I followed Sutton's notation. So, denoted policy as $\pi(a|s)$. For your case, $\pi(a|s) = \pi(s|a)$. But I think this is a mistake in this slide, it should be $\pi(a|s)$.  
  > **username_0**: I think you've denoted the policy by $\pi(s|a)$ in your answer in multiple places - or is that something else? The slide shows $a|s$  
  > **username_1**: For average reward setting $Q(s,a) = r(s,a) - r(\pi) + \sum P_{ss'}v(s')$. See Sutton's book chapter on policy funciton approximation section average reward.  
  > **username_1**: Sorry, it should be $\pi(a|s)$. It was my mistake.  
  > **username_0**: How is $\sum_{s^{\prime}}\mu(s^{\prime})\nabla_{\theta}V(s^{\prime}) = \sum_{s}\mu(s) \sum_a \pi(s,a) \sum_{s^{\prime}}P_{ss^{\prime}}^{a}\nabla_{\theta}V(s^{\prime})$? Also which edition of Sutton's book are you using?  
  > **username_1**: Because this is the product rule of probability. This is not in the Sutton's book. I give the reference, check it for more details.  
  > **username_1**: If you open the summation you can see clearly this is exactly telling us what is the probability of the agent to go to the state $s'$ from the state $s$.  
  > **username_0**: Looks good! Lastly, in places where you've written $\pi(a,s)$, did you mean $\pi(a|s)$ there too?  
  > **username_1**: Yes, I make a mistake, it should be $\pi(a|s)$.  
  > **username_0**: Your formulation ends with $Q(s,a)$, while the slide has $R(s,a)$ - why the difference? I understand the rest! It'd be great if you could elaborate on this!  
  > **username_1**: I do not follow this lecture for RL, So, I can not tell it now, first I should watch Deep RL Bootcamp. But I hope, if you think about it you may find the reason.  
  > **username_1**: Okay, I watched their lecture. At the time 40.55 they, replace $\sum r(s,a)$ with $Q(s,a)$.  
  > **username_2**: $Q(s,a)$ is an unbiased estimate of $R(s,a)$ — ie they are the same in expectation. @cogito_ai  
  
  ---
  
  ### Answer by username_2 on 2020-08-27. Score: 3
  
  The reason you are confused is because this is not the full derivation of the Policy Gradient Theorem. You are correct in thinking that $\mu(s)$ depends on the policy $\pi$ which in turn depends on the policy parameters $\theta$, and so there *should* be a derivative of $\mu$ wrt $\theta$, however the Policy Gradient Theorem doesn't require you to take this derivative.
  
  In fact, the great thing about the Policy Gradient Theorem is that the final result *does not* require you to take a derivative of the state distribution with respect to the policy parameters. I would encourage you to read and go through the derivation of the Policy Gradient Theorem from e.g. Sutton and Barto to see why you don't need to take the derivative.
  
  [![enter image description here](https://i.sstatic.net/5IN1C.png)](https://i.sstatic.net/5IN1C.png)
  
  Above is an image of the Policy Gradient Theorem proof from the Sutton and Barto book. If you carefully go through this line by line you will see that you are not required to take a derivative of the state distribution anywhere in the proof.
  
  > **Comments:**
  > **username_0**: Could you please elaborate and possibly provide mathematical details? I've gone through the derivation in Sutton and Barto and I'm still unclear why the derivation above works.  
  > **username_2**: There’s not much to elaborate on. If you followed the Sutton and Barto proof you would see that you don’t need to take a derivative of the state distribution. I can’t really explain any clearer than just writing out the full proof which I am not going to do since that is not the point of this site. If you have a particular question about the proof that you find unclear then you should post that and I’d be happy to answer it.  
  > **username_2**: Also, the above isn't a derivation. It is an extremely poor example, especially if they have not previously gone through the policy gradient theorem proof. What they have written (first two lines) gets the right answer but goes about it the wrong way.  
  > **username_0**: I **understand** the proof written in Sutton and Barto's text. My question was - **why does the wrong proof (in my post) work?** I apologize if you've already explained this, but I still don't see why the wrong proof works. Sutton and Barto's proof doesn't require us to take the derivative of the state distribution anywhere, but *that is no reason for me to overlook the same* in the proof I've attached in my post. Please explain. Thanks!  
  > **username_2**: What you have posted _is not a proof_. They have _started_ with the final result of the Policy Gradient Theorem, and then they have derived the update that is used in REINFORCE. As I say, the source you are using does not look like something I would recommend. The equality on the first line should be replaced directly with the equality on the second line to avoid confusion. Regardless, if you want good source material for Reinforcement Learning then I would recommend you read Sutton and Barto to prepare you for more advanced topics you will find in papers.  
  > **username_2**: @cogito_ai to answer your edit, what you have posted _is not a proof of the policy gradient theorem_. The author has _started_ with the final result in the policy gradient theorem, and has _derived_ something entirely separate.  
  
  ---
  
  