Answers.Body: <p>After reviewing Nan's lecture notes above, here are my conclusions</p>
  <p>let <span class="math-container">$d_t^\pi$</span> denote the distribution over <span class="math-container">$s_t$</span> and <span class="math-container">$a_t$</span> induced by policy <span class="math-container">$\pi$</span> from the initial state distribution <span class="math-container">$d_0$</span>, the first form is formally expressed as
  <span class="math-container">\begin{equation}
  \nabla J(\pi)=\frac{1}{1-\gamma}\mathbb{E}_{(s,a)\thicksim d^\pi}\left[(\nabla\log\pi(a|s))Q^\pi(s,a)\right].
  \end{equation}</span>
  where <span class="math-container">$d^\pi$</span> is the normalized state-action distribution shown below
  <span class="math-container">\begin{equation}
  d^\pi = (1 - \gamma) \sum_{t' = 1}^{\infty} \gamma^{t' - 1} d_{\pi}^{t'}
  \end{equation}</span>
  We can see that this form sums over timestamps in state-action distribution <span class="math-container">$d^\pi$</span>, while the second form sums inside the expectation. They capture long-term information in different ways.</p>
  <p>Besides, since <span class="math-container">$\frac{1}{1-\gamma}$</span> is a constant, we can ignore it and treat it as a part of learning rate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-29T04:19:25.893Z
Answers.LastActivityDate: 2024-12-29T06:36:15.417Z
Answers.LastEditDate: 2024-12-29T06:36:15.417Z
Answers.UserDisplayName: Hanqing Yu
Body: <p>I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book &quot;Reinforcement Learning: an Introduction (Sutton &amp; Barto Chapter 13)&quot;, we get the following policy gradient:
  <span class="math-container">$$
  \nabla J(\theta) = \mathbb E_\pi\left[G_t\nabla\log\pi(A_t | S_t, \theta)\right].
  $$</span>
  As we can observe from the equation, it does not relate to <strong>trajectory distributions</strong>. However, a more intuitive and widely-used introduction to policy gradient starts from defining the distribution of trajectories: <span class="math-container">$p(\tau)$</span>. For example, in <a href="https://spinningup.openai.com/en/latest/algorithms/vpg.html#key-equations" rel="nofollow noreferrer">OpenAI Spinning Up</a>, the policy gradient has the form similar to the following equation:
  <span class="math-container">$$
  \nabla J(\theta) = \mathbb E_{\tau \sim \pi}\left[\sum_{t=0}^{T}G_t\nabla_\theta\log\pi_\theta(a_t | s_t)\right].
  $$</span>
  The confusion comes from the fact that the first policy gradient does not have a summation over timestamps and is not sampling from trajectories, but the second samples from trajectories and has a summation.</p>
  <p>I did find some relevant questions about this confusion, but none of them seemed to have a good answer. Also, I could not identify any source that explained the difference/connection between the two forms.</p>
  <p>My question is why are there two different ways to describe the policy gradient and are the two forms mathematically equivalent?</p>
  <h2>Update</h2>
  <p>I found a great RL theory book (draft) written by some expert Professors in this field that shows two different formulations: <a href="https://rltheorybook.github.io" rel="nofollow noreferrer">https://rltheorybook.github.io</a>. Also, one of Nan's lecture notes shows the correlation between the two forms: <a href="https://nanjiang.cs.illinois.edu/files/cs598/note6.pdf" rel="nofollow noreferrer">https://nanjiang.cs.illinois.edu/files/cs598/note6.pdf</a>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-17T07:02:10.787Z
Comments.Text: I also have a bit of trouble understanding this, because the wsummation also appears in the GAE paper. Could you help me understand how they are the same? Which page in the RL theory book are you referring to?
Comments.UserDisplayName: craaaft
Tags: <reinforcement-learning><deep-learning><deep-rl><policy-gradients><policy-gradient-theorem>
Title: Why policy gradient theorem has two different forms?
UserDisplayName: Yuxiang Wei
ThreadText: # Why policy gradient theorem has two different forms?
  
  (Asked by: username_0 on 2023-09-23)
  
  I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton & Barto Chapter 13)", we get the following policy gradient:
  $$
  \nabla J(\theta) = \mathbb E\_\pi\left[G\_t\nabla\log\pi(A\_t | S\_t, \theta)\right].
  $$
  As we can observe from the equation, it does not relate to **trajectory distributions**. However, a more intuitive and widely-used introduction to policy gradient starts from defining the distribution of trajectories: $p(\tau)$. For example, in [OpenAI Spinning Up](https://spinningup.openai.com/en/latest/algorithms/vpg.html#key-equations), the policy gradient has the form similar to the following equation:
  $$
  \nabla J(\theta) = \mathbb E\_{\tau \sim \pi}\left[\sum\_{t=0}^{T}G\_t\nabla\_\theta\log\pi\_\theta(a\_t | s\_t)\right].
  $$
  The confusion comes from the fact that the first policy gradient does not have a summation over timestamps and is not sampling from trajectories, but the second samples from trajectories and has a summation.
  
  I did find some relevant questions about this confusion, but none of them seemed to have a good answer. Also, I could not identify any source that explained the difference/connection between the two forms.
  
  My question is why are there two different ways to describe the policy gradient and are the two forms mathematically equivalent?
  
  ## Update
  
  I found a great RL theory book (draft) written by some expert Professors in this field that shows two different formulations: <https://rltheorybook.github.io>. Also, one of Nan's lecture notes shows the correlation between the two forms: <https://nanjiang.cs.illinois.edu/files/cs598/note6.pdf>.
  
  **Tags:** reinforcement-learning, deep-learning, deep-rl, policy-gradients, policy-gradient-theorem
  > **Comments:**
  > **username_1**: I also have a bit of trouble understanding this, because the wsummation also appears in the GAE paper. Could you help me understand how they are the same? Which page in the RL theory book are you referring to?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-12-29. Score: 0
  
  After reviewing Nan's lecture notes above, here are my conclusions
  
  let $d\_t^\pi$ denote the distribution over $s\_t$ and $a\_t$ induced by policy $\pi$ from the initial state distribution $d\_0$, the first form is formally expressed as
  \begin{equation}
  \nabla J(\pi)=\frac{1}{1-\gamma}\mathbb{E}\_{(s,a)\thicksim d^\pi}\left[(\nabla\log\pi(a|s))Q^\pi(s,a)\right].
  \end{equation}
  where $d^\pi$ is the normalized state-action distribution shown below
  \begin{equation}
  d^\pi = (1 - \gamma) \sum\_{t' = 1}^{\infty} \gamma^{t' - 1} d\_{\pi}^{t'}
  \end{equation}
  We can see that this form sums over timestamps in state-action distribution $d^\pi$, while the second form sums inside the expectation. They capture long-term information in different ways.
  
  Besides, since $\frac{1}{1-\gamma}$ is a constant, we can ignore it and treat it as a part of learning rate.
  
  
  ---
  
  