<p>I'm also a newbie in the field, so I just want to share my view.</p>
<p>When I was understanding this problem, I divided the problem into two scenarios. <strong>Firstly</strong>, the scenario where we utilize a linear policy approximator and must manually compute its gradient. <strong>Secondly</strong>, the case where we employ a deep policy neural network within a framework that automatically calculates the derivative.</p>
<ul>
<li><p>In the <strong>first</strong> case, PGT enables us to calculate the gradient without touching the derivative of stationary distribution <span class="math-container">$d^\pi(s) = \lim_{t \to \infty} P(s_t = s \vert s_0, \pi_\theta)$</span>, which is affected by the policy and the environment. When the environment is unknown, calculating the derivative of this equation can be tricky.</p>
</li>
<li><p>In the <strong>second</strong> case, originally, the objective function should be <span class="math-container">$J(\theta^\pi) = \mathbb{E}_{\tau \sim \pi_\theta} [R(\tau)]$</span>, which can be used to evaluate the performance of the policy <span class="math-container">$\pi_\theta$</span>. The PGT tells us that the gradient of this equation is
<span class="math-container">$$
\nabla_\theta J(\theta^\pi) = \mathbb{E}_{s \sim d^\pi(s), a \sim \pi_\theta} [Q^\pi(s, a) \nabla_\theta \ln \pi_\theta(a \vert s)]
$$</span>
Therefore, we leverage on that to build the loss function we used to train the neural network as:
<span class="math-container">$$
\mathbb{L}(\theta^\pi) = - \mathbb{E}_{s \sim d^\pi(s), a \sim \pi_\theta} [Q^\pi(s, a) \ln \pi_\theta(a \vert s)]
$$</span>
Note that this loss function does not approximate <span class="math-container">$J(\theta^\pi) = \mathbb{E}_{\tau \sim \pi_\theta} [R(\tau)]$</span>. This loss function is only useful because it has a negative gradient of the performance when evaluated at the current parameters. &quot;<em>This means that minimizing this 'loss' function, for a given batch of data, has no guarantee whatsoever of improving expected return.</em>&quot; (Ref: <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#implementing-the-simplest-policy-gradient" rel="nofollow noreferrer">Spinning Up Doc</a>)</p>
</li>
</ul>
<p><strong>In summary</strong>, when you want to manually calculate the derivative of your <span class="math-container">$J$</span> function, PGT can be practically involved. When you use a deep learning network, PGT is also important by providing you with theoretical support of how to construct your loss function.</p>
<p>Reference and symbol denotation:</p>
<ol>
<li>Lilian Weng's <a href="https://lilianweng.github.io/posts/2018-04-08-policy-gradient/" rel="nofollow noreferrer">blog</a></li>
<li>Spinning Up <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#" rel="nofollow noreferrer">Doc</a></li>
</ol>

CC BY-SA 4.0
2024-08-03T13:49:49.023Z
Thanks for the answer, but why would the calculation of the derivative of the stastionary distribution be tricky?
craaaft
CC BY-SA 4.0
2024-08-05T09:50:14.793Z
I think that is because we normally don't know that probability distribution as it is part of the model. And in a lot of times, we use model-free approaches meaning that we don't have to know it. I hope this answers your question. @craaaft
Jiangpeng Li
CC BY-SA 4.0
2024-06-25T09:36:31.117Z
2024-07-09T05:53:30.113Z
2024-07-09T05:53:30.113Z
Jiangpeng Li
<p>What is the problem that the policy gradient solves? From what I understand the problem is taking the gradient of the state distirbution <span class="math-container">$d^{\pi_{\theta}}$</span>, but what is exactly the problem here (maybe it is a practical problem more than a theoretical one)? After applying the PGT we don't have this issue anymore. Thanks for any explanation!</p>

CC BY-SA 4.0
2024-05-29T07:51:45.730Z
you want to take the gradient of the expected sum of rewards wrt the policy, but it's not differentiable, and PGT gives you an estimator for such loss
Alberto
CC BY-SA 4.0
2024-06-02T09:24:32.927Z
MAybe stupid question, but how do I see that this is not differentiable?
craaaft
CC BY-SA 4.0
2024-06-02T10:48:56.940Z
it’s not “easily differentiable” in this case, because there is a covariance between the policy and the gradient… check this blogpost https://lilianweng.github.io/posts/2018-04-08-policy-gradient/
Alberto
<reinforcement-learning><math><policy-gradients>
Why is policy gradient theorem so important?
craaaft
# Why is policy gradient theorem so important?

(Asked by: username_0 on 2024-05-29)

What is the problem that the policy gradient solves? From what I understand the problem is taking the gradient of the state distirbution $d^{\pi\_{\theta}}$, but what is exactly the problem here (maybe it is a practical problem more than a theoretical one)? After applying the PGT we don't have this issue anymore. Thanks for any explanation!

**Tags:** reinforcement-learning, math, policy-gradients

> **Comments:**
> **username_1**: you want to take the gradient of the expected sum of rewards wrt the policy, but it's not differentiable, and PGT gives you an estimator for such loss  
> **username_0**: MAybe stupid question, but how do I see that this is not differentiable?  
> **username_1**: it’s not “easily differentiable” in this case, because there is a covariance between the policy and the gradient… check this blogpost https://lilianweng.github.io/posts/2018-04-08-policy-gradient/  

---

## 1 answers

### Answer by username_2 on 2024-06-25. Score: 2 | Accepted answer

I'm also a newbie in the field, so I just want to share my view.

When I was understanding this problem, I divided the problem into two scenarios. **Firstly**, the scenario where we utilize a linear policy approximator and must manually compute its gradient. **Secondly**, the case where we employ a deep policy neural network within a framework that automatically calculates the derivative.

* In the **first** case, PGT enables us to calculate the gradient without touching the derivative of stationary distribution $d^\pi(s) = \lim\_{t \to \infty} P(s\_t = s \vert s\_0, \pi\_\theta)$, which is affected by the policy and the environment. When the environment is unknown, calculating the derivative of this equation can be tricky.
* In the **second** case, originally, the objective function should be $J(\theta^\pi) = \mathbb{E}\_{\tau \sim \pi\_\theta} [R(\tau)]$, which can be used to evaluate the performance of the policy $\pi\_\theta$. The PGT tells us that the gradient of this equation is
  $$
  \nabla\_\theta J(\theta^\pi) = \mathbb{E}\_{s \sim d^\pi(s), a \sim \pi\_\theta} [Q^\pi(s, a) \nabla\_\theta \ln \pi\_\theta(a \vert s)]
  $$
  Therefore, we leverage on that to build the loss function we used to train the neural network as:
  $$
  \mathbb{L}(\theta^\pi) = - \mathbb{E}\_{s \sim d^\pi(s), a \sim \pi\_\theta} [Q^\pi(s, a) \ln \pi\_\theta(a \vert s)]
  $$
  Note that this loss function does not approximate $J(\theta^\pi) = \mathbb{E}\_{\tau \sim \pi\_\theta} [R(\tau)]$. This loss function is only useful because it has a negative gradient of the performance when evaluated at the current parameters. "*This means that minimizing this 'loss' function, for a given batch of data, has no guarantee whatsoever of improving expected return.*" (Ref: [Spinning Up Doc](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#implementing-the-simplest-policy-gradient))

**In summary**, when you want to manually calculate the derivative of your $J$ function, PGT can be practically involved. When you use a deep learning network, PGT is also important by providing you with theoretical support of how to construct your loss function.

Reference and symbol denotation:

1. Lilian Weng's [blog](https://lilianweng.github.io/posts/2018-04-08-policy-gradient/)
2. Spinning Up [Doc](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#)

> **Comments:**
> **username_0**: Thanks for the answer, but why would the calculation of the derivative of the stastionary distribution be tricky?  
> **username_2**: I think that is because we normally don't know that probability distribution as it is part of the model. And in a lot of times, we use model-free approaches meaning that we don't have to know it. I hope this answers your question. @username_0  

---

