Answers.Body: <p>I'm also a newbie in the field, so I just want to share my view.</p>
  <p>When I was understanding this problem, I divided the problem into two scenarios. <strong>Firstly</strong>, the scenario where we utilize a linear policy approximator and must manually compute its gradient. <strong>Secondly</strong>, the case where we employ a deep policy neural network within a framework that automatically calculates the derivative.</p>
  <ul>
  <li><p>In the <strong>first</strong> case, PGT enables us to calculate the gradient without touching the derivative of stationary distribution <span class="math-container">$d^\pi(s) = \lim_{t \to \infty} P(s_t = s \vert s_0, \pi_\theta)$</span>, which is affected by the policy and the environment. When the environment is unknown, calculating the derivative of this equation can be tricky.</p>
  </li>
  <li><p>In the <strong>second</strong> case, originally, the objective function should be <span class="math-container">$J(\theta^\pi) = \mathbb{E}_{\tau \sim \pi_\theta} [R(\tau)]$</span>, which can be used to evaluate the performance of the policy <span class="math-container">$\pi_\theta$</span>. The PGT tells us that the gradient of this equation is
  <span class="math-container">$$
  \nabla_\theta J(\theta^\pi) = \mathbb{E}_{s \sim d^\pi(s), a \sim \pi_\theta} [Q^\pi(s, a) \nabla_\theta \ln \pi_\theta(a \vert s)]
  $$</span>
  Therefore, we leverage on that to build the loss function we used to train the neural network as:
  <span class="math-container">$$
  \mathbb{L}(\theta^\pi) = - \mathbb{E}_{s \sim d^\pi(s), a \sim \pi_\theta} [Q^\pi(s, a) \ln \pi_\theta(a \vert s)]
  $$</span>
  Note that this loss function does not approximate <span class="math-container">$J(\theta^\pi) = \mathbb{E}_{\tau \sim \pi_\theta} [R(\tau)]$</span>. This loss function is only useful because it has a negative gradient of the performance when evaluated at the current parameters. &quot;<em>This means that minimizing this 'loss' function, for a given batch of data, has no guarantee whatsoever of improving expected return.</em>&quot; (Ref: <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#implementing-the-simplest-policy-gradient" rel="nofollow noreferrer">Spinning Up Doc</a>)</p>
  </li>
  </ul>
  <p><strong>In summary</strong>, when you want to manually calculate the derivative of your <span class="math-container">$J$</span> function, PGT can be practically involved. When you use a deep learning network, PGT is also important by providing you with theoretical support of how to construct your loss function.</p>
  <p>Reference and symbol denotation:</p>
  <ol>
  <li>Lilian Weng's <a href="https://lilianweng.github.io/posts/2018-04-08-policy-gradient/" rel="nofollow noreferrer">blog</a></li>
  <li>Spinning Up <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#" rel="nofollow noreferrer">Doc</a></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-03T13:49:49.023Z
Answers.Text: Thanks for the answer, but why would the calculation of the derivative of the stastionary distribution be tricky?
Answers.UserDisplayName: craaaft
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-05T09:50:14.793Z
Answers.Text: I think that is because we normally don't know that probability distribution as it is part of the model. And in a lot of times, we use model-free approaches meaning that we don't have to know it. I hope this answers your question. @craaaft
Answers.UserDisplayName: Jiangpeng Li
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-25T09:36:31.117Z
Answers.LastActivityDate: 2024-07-09T05:53:30.113Z
Answers.LastEditDate: 2024-07-09T05:53:30.113Z
Answers.UserDisplayName: Jiangpeng Li
Body: <p>What is the problem that the policy gradient solves? From what I understand the problem is taking the gradient of the state distirbution <span class="math-container">$d^{\pi_{\theta}}$</span>, but what is exactly the problem here (maybe it is a practical problem more than a theoretical one)? After applying the PGT we don't have this issue anymore. Thanks for any explanation!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-29T07:51:45.730Z
Comments.Text: you want to take the gradient of the expected sum of rewards wrt the policy, but it's not differentiable, and PGT gives you an estimator for such loss
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-02T09:24:32.927Z
Comments.Text: MAybe stupid question, but how do I see that this is not differentiable?
Comments.UserDisplayName: craaaft
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-02T10:48:56.940Z
Comments.Text: it’s not “easily differentiable” in this case, because there is a covariance between the policy and the gradient… check this blogpost https://lilianweng.github.io/posts/2018-04-08-policy-gradient/
Comments.UserDisplayName: Alberto
Tags: <reinforcement-learning><math><policy-gradients>
Title: Why is policy gradient theorem so important?
UserDisplayName: craaaft
ThreadText: # Why is policy gradient theorem so important?
  
  (Asked by: username_0 on 2024-05-29)
  
  What is the problem that the policy gradient solves? From what I understand the problem is taking the gradient of the state distirbution $d^{\pi\_{\theta}}$, but what is exactly the problem here (maybe it is a practical problem more than a theoretical one)? After applying the PGT we don't have this issue anymore. Thanks for any explanation!
  
  **Tags:** reinforcement-learning, math, policy-gradients
  
  > **Comments:**
  > **username_1**: you want to take the gradient of the expected sum of rewards wrt the policy, but it's not differentiable, and PGT gives you an estimator for such loss  
  > **username_0**: MAybe stupid question, but how do I see that this is not differentiable?  
  > **username_1**: it’s not “easily differentiable” in this case, because there is a covariance between the policy and the gradient… check this blogpost https://lilianweng.github.io/posts/2018-04-08-policy-gradient/  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-06-25. Score: 2 | Accepted answer
  
  I'm also a newbie in the field, so I just want to share my view.
  
  When I was understanding this problem, I divided the problem into two scenarios. **Firstly**, the scenario where we utilize a linear policy approximator and must manually compute its gradient. **Secondly**, the case where we employ a deep policy neural network within a framework that automatically calculates the derivative.
  
  * In the **first** case, PGT enables us to calculate the gradient without touching the derivative of stationary distribution $d^\pi(s) = \lim\_{t \to \infty} P(s\_t = s \vert s\_0, \pi\_\theta)$, which is affected by the policy and the environment. When the environment is unknown, calculating the derivative of this equation can be tricky.
  * In the **second** case, originally, the objective function should be $J(\theta^\pi) = \mathbb{E}\_{\tau \sim \pi\_\theta} [R(\tau)]$, which can be used to evaluate the performance of the policy $\pi\_\theta$. The PGT tells us that the gradient of this equation is
    $$
    \nabla\_\theta J(\theta^\pi) = \mathbb{E}\_{s \sim d^\pi(s), a \sim \pi\_\theta} [Q^\pi(s, a) \nabla\_\theta \ln \pi\_\theta(a \vert s)]
    $$
    Therefore, we leverage on that to build the loss function we used to train the neural network as:
    $$
    \mathbb{L}(\theta^\pi) = - \mathbb{E}\_{s \sim d^\pi(s), a \sim \pi\_\theta} [Q^\pi(s, a) \ln \pi\_\theta(a \vert s)]
    $$
    Note that this loss function does not approximate $J(\theta^\pi) = \mathbb{E}\_{\tau \sim \pi\_\theta} [R(\tau)]$. This loss function is only useful because it has a negative gradient of the performance when evaluated at the current parameters. "*This means that minimizing this 'loss' function, for a given batch of data, has no guarantee whatsoever of improving expected return.*" (Ref: [Spinning Up Doc](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#implementing-the-simplest-policy-gradient))
  
  **In summary**, when you want to manually calculate the derivative of your $J$ function, PGT can be practically involved. When you use a deep learning network, PGT is also important by providing you with theoretical support of how to construct your loss function.
  
  Reference and symbol denotation:
  
  1. Lilian Weng's [blog](https://lilianweng.github.io/posts/2018-04-08-policy-gradient/)
  2. Spinning Up [Doc](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#)
  
  > **Comments:**
  > **username_0**: Thanks for the answer, but why would the calculation of the derivative of the stastionary distribution be tricky?  
  > **username_2**: I think that is because we normally don't know that probability distribution as it is part of the model. And in a lot of times, we use model-free approaches meaning that we don't have to know it. I hope this answers your question. @username_0  
  
  ---
  
  