Answers.Body: <p>You cannot do this:</p>
  
  <blockquote>
    <p><span class="math-container">$\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\bigtriangledown log \pi (\tau )]$</span></p>
  </blockquote>
  
  <p>That is because <span class="math-container">$r(\tau )$</span> and <span class="math-container">$\bigtriangledown log \pi (\tau )$</span> are correlated by their dependence on <span class="math-container">$\tau$</span>. In a simpler concrete example, if your expectation was over simple equiprobable discrete distribution where <span class="math-container">$\tau$</span> could be any integer in range <span class="math-container">$[1,10]$</span>, then <span class="math-container">$\mathop{\mathbb{E} }[\tau^2] = 38.5$</span> whilst <span class="math-container">$\mathop{\mathbb{E} }[\tau]\mathop{\mathbb{E} }[\tau] = 30.25$</span></p>
  
  <blockquote>
    <p>The  pytorch implementation simply multiplied log probability and reward <code>-log_prob * R</code> and then summed the vector <code>torch.cat(policy_loss).sum()</code> there is no <span class="math-container">$p(\tau)$</span>. What is really happening here?</p>
  </blockquote>
  
  <p>The purpose of transforming the gradient into an <em>expectation</em> <span class="math-container">$\mathbb{E}$</span> for the policy gradient theorem, is so that you can estimate it using <em>samples</em> taken from the distribution. Typically, you don't know <span class="math-container">$p(\tau)$</span>, but you do know that if you follow the same process where <span class="math-container">$p(\tau)$</span> applies (i.e. measure the return from the environment whilst following the policy represented by the policy function) that you will get an unbiased <em>sample</em> from that distribution.</p>
  
  <p>So what is going on here is that you throw away the outer expectation <span class="math-container">$\mathbb{E}_{\pi}[]$</span> and replace it with a stochastic estimate for the same value based on taking samples. The samples are naturally obtained with distribution described by <span class="math-container">$p(\tau)$</span>, if you follow the policy function when making action choices.</p>
  
  <blockquote>
    <p>The second question is the multiplication of log probability and reward in pytorch implementation <code>-log_prob * R</code>, pytorch implementation has a <strong>negative</strong> log probability and derived equation has a positive one <span class="math-container">$\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )]$</span>. What is the need for multipling log probability with negative value in pytorch implementaion?</p>
  </blockquote>
  
  <p>I don't know the code, but this very likely because of a sign change brought on by considering how to respond to the gradient estimate. </p>
  
  <p>There is a clue in the use of the name "loss". To maximise return in policy gradient methods, you can perform gradient <em>ascent</em> based on the estimated gradient as the goal is to find higher values. However, it is more usual in NN libraries to perform gradient <em>descent</em> in order to minimise a loss function. That is a likely cause of the sign reversal here.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-08T14:19:04.187Z
Answers.Text: `you throw away the outer expectation Eπ[]Eπ[] and replace it with a stochastic estimate for the same value based on taking samples.` When I see  an expectation equation such as this $\mathop{\mathbb{E}}[f(x)]$ I always consider it as an "average" of that function. When replacing it with stochastic estimate  are we now calculating the "total" instead of the "average"?  Like we see in the code `policy_loss = torch.cat(policy_loss).sum()`
Answers.UserDisplayName: Eka
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-08T14:21:56.383Z
Answers.Text: @eka: It will still be an average, but averaged over the samples you took instead of over a distribution function that you (probably) don't know. Although the code you show does indeed caclulate a total. If the batch size is always the same, this is just a scale difference though, and might be corrected for in later code, or in the learning rate etc.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-08T14:27:30.080Z
Answers.Text: I am really weak in the area of math and I really appreciate that you took your time to explain my queries. Thank you
Answers.UserDisplayName: Eka
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-08T08:24:04.223Z
Answers.LastActivityDate: 2019-11-08T13:53:37.633Z
Answers.LastEditDate: 2019-11-08T13:53:37.633Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am trying to understand the policy gradient method using <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py" rel="nofollow noreferrer">a PyTorch implementation</a> and <a href="https://towardsdatascience.com/policy-gradients-in-a-nutshell-8b72f9743c5d" rel="nofollow noreferrer">this tutorial</a>.</p>
  
  <p>My first question is about the end result of this gradient derivation,</p>
  
  <p><span class="math-container">\begin{aligned}
  \nabla \mathbb{E}_{\pi}[r(\tau)] &amp;=\nabla \int \pi(\tau) r(\tau) d \tau \\
  &amp;=\int \nabla \pi(\tau) r(\tau) d \tau \\
  &amp;=\int \pi(\tau) \nabla \log \pi(\tau) r(\tau) d \tau \\
  \nabla \mathbb{E}_{\pi}[r(\tau)] &amp;=\mathbb{E}_{\pi}[r(\tau) \nabla \log \pi(\tau)]
  \end{aligned}</span></p>
  
  <p>Mainly in this equation </p>
  
  <p><span class="math-container">$$\nabla \mathop{\mathbb{E}_\pi }[r(\tau )] = \mathop{\mathbb{E}_\pi }[r(\tau )\nabla log \pi (\tau )]$$</span></p>
  
  <p>Does expectation follow a distributive or associative property?</p>
  
  <p>I know that expectations of a function can be written as below</p>
  
  <p><span class="math-container">$$\mathop{\mathbb{E}}[f(x)] =\sum p(x)f(x)$$</span></p>
  
  <p>Then can we rewrite the first equations as</p>
  
  <p><span class="math-container">$$\mathop{\mathbb{E}_\pi }[r(\tau )\nabla log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\nabla log \pi (\tau )] \\= \sum p(\tau)r(\tau ) \,\, \sum p(\tau)\nabla log \pi (\tau ) \\
  = p(\tau) \sum r(\tau ) \nabla log \pi (\tau )$$</span></p>
  
  <p>The problem is when I compare  this to PyTorch implementation (line 71-74)</p>
  
  <pre><code>for log_prob, R in zip(policy.saved_log_probs, returns):
      policy_loss.append(-log_prob * R)
  optimizer.zero_grad()
  policy_loss = torch.cat(policy_loss).sum()
  </code></pre>
  
  <p>The  pytorch implementation simply multiplied log probability and reward <code>-log_prob * R</code> and then summed the vector <code>torch.cat(policy_loss).sum()</code> there is no <span class="math-container">$p(\tau)$</span>. What is really happening here?</p>
  
  <p>The second question is the multiplication of log probability and reward in PyTorch implementation <code>-log_prob * R</code>, PyTorch implementation has a <strong>negative</strong> log probability and derived equation has a positive one <span class="math-container">$\mathop{\mathbb{E}_\pi }[r(\tau )\nabla log \pi (\tau )]$</span>. What is the need for multiplying log probability with a negative value in PyTorch implementation?</p>
  
  <p>I have only a basic understanding of maths and that's why I am asking this question here.</p>
  
  <hr>
  
  <p>Edit: found a better derivation of above equation <a href="https://youtu.be/Ys3YY7sSmIA?t=3622" rel="nofollow noreferrer">https://youtu.be/Ys3YY7sSmIA?t=3622</a></p>
  
Tags: <reinforcement-learning><policy-gradients><pytorch><reinforce><expectation>
Title: How does the policy gradient's derivative work?
UserDisplayName: Eka
ThreadText: # How does the policy gradient's derivative work?
  
  (Asked by: username_0 on 2019-11-08)
  
  I am trying to understand the policy gradient method using [a PyTorch implementation](https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py) and [this tutorial](https://towardsdatascience.com/policy-gradients-in-a-nutshell-8b72f9743c5d).
  
  My first question is about the end result of this gradient derivation,
  
  \begin{aligned}
  \nabla \mathbb{E}\_{\pi}[r(\tau)] &=\nabla \int \pi(\tau) r(\tau) d \tau \\
  &=\int \nabla \pi(\tau) r(\tau) d \tau \\
  &=\int \pi(\tau) \nabla \log \pi(\tau) r(\tau) d \tau \\
  \nabla \mathbb{E}\_{\pi}[r(\tau)] &=\mathbb{E}\_{\pi}[r(\tau) \nabla \log \pi(\tau)]
  \end{aligned}
  
  Mainly in this equation
  
  $$\nabla \mathop{\mathbb{E}\_\pi }[r(\tau )] = \mathop{\mathbb{E}\_\pi }[r(\tau )\nabla log \pi (\tau )]$$
  
  Does expectation follow a distributive or associative property?
  
  I know that expectations of a function can be written as below
  
  $$\mathop{\mathbb{E}}[f(x)] =\sum p(x)f(x)$$
  
  Then can we rewrite the first equations as
  
  $$\mathop{\mathbb{E}\_\pi }[r(\tau )\nabla log \pi (\tau )] \\= \mathop{\mathbb{E}\_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}\_\pi }[\nabla log \pi (\tau )] \\= \sum p(\tau)r(\tau ) \,\, \sum p(\tau)\nabla log \pi (\tau ) \\
  = p(\tau) \sum r(\tau ) \nabla log \pi (\tau )$$
  
  The problem is when I compare this to PyTorch implementation (line 71-74)
  
  ```
  for log_prob, R in zip(policy.saved_log_probs, returns):
      policy_loss.append(-log_prob * R)
  optimizer.zero_grad()
  policy_loss = torch.cat(policy_loss).sum()
  
  ```
  
  The pytorch implementation simply multiplied log probability and reward `-log_prob * R` and then summed the vector `torch.cat(policy_loss).sum()` there is no $p(\tau)$. What is really happening here?
  
  The second question is the multiplication of log probability and reward in PyTorch implementation `-log_prob * R`, PyTorch implementation has a **negative** log probability and derived equation has a positive one $\mathop{\mathbb{E}\_\pi }[r(\tau )\nabla log \pi (\tau )]$. What is the need for multiplying log probability with a negative value in PyTorch implementation?
  
  I have only a basic understanding of maths and that's why I am asking this question here.
  
  ---
  
  Edit: found a better derivation of above equation <https://youtu.be/Ys3YY7sSmIA?t=3622>
  
  **Tags:** reinforcement-learning, policy-gradients, pytorch, reinforce, expectation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-08. Score: 2 | Accepted answer
  
  You cannot do this:
  
  > $\mathop{\mathbb{E}\_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}\_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}\_\pi }[\bigtriangledown log \pi (\tau )]$
  
  That is because $r(\tau )$ and $\bigtriangledown log \pi (\tau )$ are correlated by their dependence on $\tau$. In a simpler concrete example, if your expectation was over simple equiprobable discrete distribution where $\tau$ could be any integer in range $[1,10]$, then $\mathop{\mathbb{E} }[\tau^2] = 38.5$ whilst $\mathop{\mathbb{E} }[\tau]\mathop{\mathbb{E} }[\tau] = 30.25$
  
  > The pytorch implementation simply multiplied log probability and reward `-log_prob * R` and then summed the vector `torch.cat(policy_loss).sum()` there is no $p(\tau)$. What is really happening here?
  
  The purpose of transforming the gradient into an *expectation* $\mathbb{E}$ for the policy gradient theorem, is so that you can estimate it using *samples* taken from the distribution. Typically, you don't know $p(\tau)$, but you do know that if you follow the same process where $p(\tau)$ applies (i.e. measure the return from the environment whilst following the policy represented by the policy function) that you will get an unbiased *sample* from that distribution.
  
  So what is going on here is that you throw away the outer expectation $\mathbb{E}\_{\pi}[]$ and replace it with a stochastic estimate for the same value based on taking samples. The samples are naturally obtained with distribution described by $p(\tau)$, if you follow the policy function when making action choices.
  
  > The second question is the multiplication of log probability and reward in pytorch implementation `-log_prob * R`, pytorch implementation has a **negative** log probability and derived equation has a positive one $\mathop{\mathbb{E}\_\pi }[r(\tau )\bigtriangledown log \pi (\tau )]$. What is the need for multipling log probability with negative value in pytorch implementaion?
  
  I don't know the code, but this very likely because of a sign change brought on by considering how to respond to the gradient estimate.
  
  There is a clue in the use of the name "loss". To maximise return in policy gradient methods, you can perform gradient *ascent* based on the estimated gradient as the goal is to find higher values. However, it is more usual in NN libraries to perform gradient *descent* in order to minimise a loss function. That is a likely cause of the sign reversal here.
  
  > **Comments:**
  > **username_0**: `you throw away the outer expectation Eπ[]Eπ[] and replace it with a stochastic estimate for the same value based on taking samples.` When I see  an expectation equation such as this $\mathop{\mathbb{E}}[f(x)]$ I always consider it as an "average" of that function. When replacing it with stochastic estimate  are we now calculating the "total" instead of the "average"?  Like we see in the code `policy_loss = torch.cat(policy_loss).sum()`  
  > **username_1**: @eka: It will still be an average, but averaged over the samples you took instead of over a distribution function that you (probably) don't know. Although the code you show does indeed caclulate a total. If the batch size is always the same, this is just a scale difference though, and might be corrected for in later code, or in the learning rate etc.  
  > **username_0**: I am really weak in the area of math and I really appreciate that you took your time to explain my queries. Thank you  
  
  ---
  
  