Answers.Body: <p>The identity <span class="math-container">$$\nabla_{\theta} P(\tau \mid \theta) = P(\tau \mid \theta) \nabla_{\theta} \log P(\tau \mid \theta)\tag{1}\label{1},$$</span> </p>
  
  <p>which can also be written as </p>
  
  <p><span class="math-container">\begin{align}
  \nabla_{\theta} \log P(\tau \mid \theta) 
  &amp;= 
  \frac{\nabla_{\theta} P(\tau \mid \theta)}{P(\tau \mid \theta)}\\
  &amp;=\frac{1}{P(\tau \mid \theta)} \nabla_{\theta} P(\tau \mid \theta)
  \end{align}</span></p>
  
  <p>directly comes from the general rule to derive the logarithm of a function and the chain rule
  <span class="math-container">\begin{align}
  \frac{d \log f(x)}{d x} 
  &amp;= \frac{1}{f(x)} \frac{d f}{dx}.
  \end{align}</span>
  Note that <span class="math-container">$\log f(x)$</span> is a composite function and that's why we apply the chain rule and that the derivative of <span class="math-container">$\log x = \frac{1}{x}$</span>, as your text says.</p>
  
  <p>People shouldn't call this a trick. There's no trick here. It's just basic calculus.</p>
  
  <p>Why do you need identity \ref{1}? Because that identity tells you that the derivative of the probability of the trajectory given the parameter <span class="math-container">$\theta$</span> with respect to <span class="math-container">$\theta$</span> is <span class="math-container">$P(\tau \mid \theta)$</span> times the gradient of the logarithm of that same probability. How is this useful? Because the logarithm will turn your product into a sum (and the derivative of a sum is the sum of the derivatives of the elements of the sum), Essentially, the identity \ref{1} will help you to compute the gradient is an easier way (at least, conceptually).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T23:30:57.830Z
Answers.Text: Thank you for your answer, it is clear how the derivative of log fits into the equation but not how the first form is rearranged to fit the second form. Are you able to elaborate on this?
Answers.UserDisplayName: Jacob B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T23:44:13.057Z
Answers.Text: @JacobB The first form is **not** rearranged to fit the second form in your example. The gradient of $P$ is given by your 2nd equation. Your 1st equation only tells you the definition of the probability of the trajectory. Then you can use your 2nd identity to find the gradient of the 1st. Why do you need the 2nd identity? Because there's a log there and your product in the 1st equation will become a sum.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T23:51:04.493Z
Answers.Text: I see thanks, I misunderstood the transition between 1&2 it seemed they were building off the last equation.
Answers.UserDisplayName: Jacob B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T23:55:23.350Z
Answers.Text: In your follow up you state we use the log of probabilities to make our computation easier, but we still have to compute the original probability don't we? So we can't completely avoid a long chain of multiplications.
Answers.UserDisplayName: Jacob B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T23:59:35.537Z
Answers.Text: @JacobB Multiplications (especially of small numbers) can easily be numerically unstable. For example, if you multiply $0.5$ and $0.5$, you get $0.25$. If you keep multiplying small numbers, you can easily go to zero _with finite-precision numbers_. In your example, if $T$ is a big number, that can easily happen. However, summations do not suffer from this (i.e. $0.5+0.5 > 0.5$). That's another reason to use that trick. But, essentially, you use that trick because computing the gradient of a summation is easier than computing the gradient of a product.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T00:14:34.673Z
Answers.Text: Right, but in the formula we don't avoid this unstable multiplication only part of the formula is contained in a log
Answers.UserDisplayName: Jacob B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-30T05:44:38.927Z
Answers.Text: The reason for using the log here is indeed to turn a product into a sum. But I don't believe it's for numerical reasons. Instead the reason is that the trajectory likelihood is a product of terms $p_\theta(a_t|s_t)p(s_{t+1}|a_t, s_t)$, with a policy $p_\theta(a_t|s_t)$ with parameters $\theta$. Once taking the log, we get a sum and the derivatives of the dynamics terms $p(s_{t+1}|a_t, s_t)$ with respect to the policy parameters $\theta$ drop out.
Answers.UserDisplayName: Chris Cundy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T23:18:19.543Z
Answers.LastActivityDate: 2020-04-26T23:55:28.240Z
Answers.LastEditDate: 2020-04-26T23:55:28.240Z
Answers.UserDisplayName: nbro
Body: <p>I am looking at this formula which breaks down the gradient of <span class="math-container">$P(\tau |\theta)$</span> the first part is clear as is the derivative of <span class="math-container">$\log(x)$</span>, but I do not see how the first formula is rearranged into the second.</p>
  
  <p><a href="https://i.sstatic.net/fuTAh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fuTAh.png" alt="enter image description here"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-30T11:01:04.407Z
Comments.Text: The screenshot comes from the [OpenAI page](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#implementing-the-simplest-policy-gradient). It's not a rearrangement or derivation. It is just a statement. $\nabla f(x) = f(x) \nabla log f(x)$. Instead of $f(x)$ you have a parametrized probability which you want to maximize. The next point (3) on that page uses this statement and applies it on the trajectory
Comments.UserDisplayName: ahron
Tags: <reinforcement-learning><math><policy-gradients><calculus>
Title: How is the log-derivative trick of a trajectory derived?
UserDisplayName: Jacob B
ThreadText: # How is the log-derivative trick of a trajectory derived?
  
  (Asked by: username_0 on 2020-04-26)
  
  I am looking at this formula which breaks down the gradient of $P(\tau |\theta)$ the first part is clear as is the derivative of $\log(x)$, but I do not see how the first formula is rearranged into the second.
  
  [![enter image description here](https://i.sstatic.net/fuTAh.png)](https://i.sstatic.net/fuTAh.png)
  
  **Tags:** reinforcement-learning, math, policy-gradients, calculus
  > **Comments:**
  > **username_1**: The screenshot comes from the [OpenAI page](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#implementing-the-simplest-policy-gradient). It's not a rearrangement or derivation. It is just a statement. $\nabla f(x) = f(x) \nabla log f(x)$. Instead of $f(x)$ you have a parametrized probability which you want to maximize. The next point (3) on that page uses this statement and applies it on the trajectory  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-04-26. Score: 3 | Accepted answer
  
  The identity $$\nabla\_{\theta} P(\tau \mid \theta) = P(\tau \mid \theta) \nabla\_{\theta} \log P(\tau \mid \theta)\tag{1}\label{1},$$
  
  which can also be written as
  
  \begin{align}
  \nabla\_{\theta} \log P(\tau \mid \theta)
  &=
  \frac{\nabla\_{\theta} P(\tau \mid \theta)}{P(\tau \mid \theta)}\\
  &=\frac{1}{P(\tau \mid \theta)} \nabla\_{\theta} P(\tau \mid \theta)
  \end{align}
  
  directly comes from the general rule to derive the logarithm of a function and the chain rule
  \begin{align}
  \frac{d \log f(x)}{d x}
  &= \frac{1}{f(x)} \frac{d f}{dx}.
  \end{align}
  Note that $\log f(x)$ is a composite function and that's why we apply the chain rule and that the derivative of $\log x = \frac{1}{x}$, as your text says.
  
  People shouldn't call this a trick. There's no trick here. It's just basic calculus.
  
  Why do you need identity \ref{1}? Because that identity tells you that the derivative of the probability of the trajectory given the parameter $\theta$ with respect to $\theta$ is $P(\tau \mid \theta)$ times the gradient of the logarithm of that same probability. How is this useful? Because the logarithm will turn your product into a sum (and the derivative of a sum is the sum of the derivatives of the elements of the sum), Essentially, the identity \ref{1} will help you to compute the gradient is an easier way (at least, conceptually).
  
  > **Comments:**
  > **username_0**: Thank you for your answer, it is clear how the derivative of log fits into the equation but not how the first form is rearranged to fit the second form. Are you able to elaborate on this?  
  > **username_2**: @JacobB The first form is **not** rearranged to fit the second form in your example. The gradient of $P$ is given by your 2nd equation. Your 1st equation only tells you the definition of the probability of the trajectory. Then you can use your 2nd identity to find the gradient of the 1st. Why do you need the 2nd identity? Because there's a log there and your product in the 1st equation will become a sum.  
  > **username_0**: I see thanks, I misunderstood the transition between 1&2 it seemed they were building off the last equation.  
  > **username_0**: In your follow up you state we use the log of probabilities to make our computation easier, but we still have to compute the original probability don't we? So we can't completely avoid a long chain of multiplications.  
  > **username_2**: @JacobB Multiplications (especially of small numbers) can easily be numerically unstable. For example, if you multiply $0.5$ and $0.5$, you get $0.25$. If you keep multiplying small numbers, you can easily go to zero _with finite-precision numbers_. In your example, if $T$ is a big number, that can easily happen. However, summations do not suffer from this (i.e. $0.5+0.5 > 0.5$). That's another reason to use that trick. But, essentially, you use that trick because computing the gradient of a summation is easier than computing the gradient of a product.  
  > **username_0**: Right, but in the formula we don't avoid this unstable multiplication only part of the formula is contained in a log  
  > **username_3**: The reason for using the log here is indeed to turn a product into a sum. But I don't believe it's for numerical reasons. Instead the reason is that the trajectory likelihood is a product of terms $p_\theta(a_t|s_t)p(s_{t+1}|a_t, s_t)$, with a policy $p_\theta(a_t|s_t)$ with parameters $\theta$. Once taking the log, we get a sum and the derivatives of the dynamics terms $p(s_{t+1}|a_t, s_t)$ with respect to the policy parameters $\theta$ drop out.  
  
  ---
  
  