Answers.Body: <p>Short answer:</p>
  <p>The expectation <span class="math-container">$\mathbb{E}_t$</span> in the PPO paper is not an expectation over trajectories, but a mean. I suppose the confusion comes from there. The two quantities are otherwise very similar. The gradient is however not the same, as it is a biased estimator.</p>
  <p>Long answer:</p>
  <p>There are many related formulations for RL problems, on one hand we have the finite-horizon (episodic) setting:</p>
  <p><span class="math-container">$$ J_1(\theta)=\mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[\sum_{t= 0}^T r(s_t,a_t)\right]. $$</span></p>
  <p>On the other hand, we have the commonly used infinite-horizon setting (assumed by the PPO paper):</p>
  <p><span class="math-container">$$ J_2(\theta)=\mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[\sum_{t\geq 0} \gamma^tr(s_t,a_t)\right]. $$</span></p>
  <p>In the finite horizon setting, we have :</p>
  <p><span class="math-container">\begin{equation} \nabla_{\theta} J_1(\theta) = \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \left(\sum_{t= 0}^T \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right)R(\tau) \right], \end{equation}</span></p>
  <p>where <span class="math-container">$R(\tau)=\sum_{t= 0}^T r(s_t,a_t)$</span>. As explained in the SpinningUp post, we can replace <span class="math-container">$R(\tau)$</span> by <span class="math-container">$A_{\pi_{\theta}}(s_t,a_t)$</span>.</p>
  <p>Now, a similar proof shows that:</p>
  <p><span class="math-container">\begin{equation} \nabla_{\theta} J_2(\theta) = \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \sum_{t\geq 0}\gamma^t \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) A_{\pi_{\theta}}(s_t,a_t) \right]. \end{equation}</span></p>
  <p>Now, in practice it is very common to approximate</p>
  <p><span class="math-container">$$\nabla_{\theta} J_2(\theta) \approx \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \sum_{t\geq 0}\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) A_{\pi_{\theta}}(s_t,a_t) \right]:=g^{\gamma},$$</span></p>
  <p>that is we drop the discount factor <span class="math-container">$\gamma^t$</span>. This relates to the policy gradient theorem and that we want to sample states from the steady-state distribution instead of the discounted state distribution.</p>
  <p>Now, we want to approximate <span class="math-container">$A_{\pi}(s_t,a_t)$</span> by an estimator <span class="math-container">$\hat{A_t}$</span>. Schulman et al in the <a href="https://arxiv.org/abs/1506.02438" rel="nofollow noreferrer">GAE paper</a> introduce the definition of a <span class="math-container">$\gamma$</span>-just estimator. The exact definition is maybe not essential, but <span class="math-container">$\gamma$</span>-just estimators include <span class="math-container">$Q_{\pi}(s_t,a_t)$</span>, <span class="math-container">$A_{\pi}(s_t,a_t)$</span> and the GAE estimator used in the PPO paper.</p>
  <p>So, if <span class="math-container">$\hat{A}_t$</span> is <span class="math-container">$\gamma$</span>-just, then we have:
  <span class="math-container">$$g^{\gamma} = \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \sum_{t\geq 0}\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \hat{A}_t \right]. $$</span></p>
  <p>This expectation is however still not convenient for a practical algorithm, as we need to collect the trajectories with a policy with older weights <span class="math-container">$\pi_{\theta_{\text{old}}}$</span>. So we approximate</p>
  <p><span class="math-container">$$g^{\gamma} \approx \mathbb{E}_{\tau\sim p_{\pi_{\theta_{\text{old}}}}}\left[ \sum_{t\geq 0}\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \hat{A}_t \right]:=\hat{g}. $$</span></p>
  <p>Now, we need to approximate this expectation, so we collect <span class="math-container">$N$</span> rollouts of identical length <span class="math-container">$T$</span> (for simplicity) with the policy <span class="math-container">$\pi_{\theta_{\text{old}}}$</span>, then we have an approximation of <span class="math-container">$g^{\gamma}$</span>:</p>
  <p><span class="math-container">$$\nabla_{\theta} J_2(\theta) \approx g^{\gamma}\approx  \hat{g}\approx\frac{1}{N}\sum_{n=0}^N\sum_{t=0}^T \nabla_{\theta}\log\pi_{\theta}(a_t^n\mid s_t^n) \hat{A}_t^n,$$</span>
  which is what the <span class="math-container">$\mathbb{E}_t$</span> in the PPO paper means. The expression on the right-hand side is now finally useful for gradient descent-like algorithms, as <span class="math-container">$\hat{A_t}$</span>does not depend on <span class="math-container">$\theta$</span>.</p>
  <p>Note also that much of the PPO paper aims to ensure that the last approximation by replacing <span class="math-container">$p_{\pi_{\theta}}$</span> by  <span class="math-container">$p_{\pi_{\theta_{\text{old}}}}$</span> is valid.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-08T16:25:46.797Z
Answers.Text: Thanks for the detailed explanation. Indeed, it does seem like the confusion stems from the $\hat{E}_{t}$ notation. I understand your derivation but I dont get why your last estimator $\hat{g}$ is what they refer to by $\hat{E}_{t}$. For me "an empirical batch over a finite batch of samples" is rather your last estimor without the sum over t, i.e. $\frac{1}{N} \sum_{n = 0}^{N} \nabla_{\theta} \log \pi_{\theta}(a_{t}^{n} \mid s_{t}^{n}) \hat{A}_{t}^{n}$.
Answers.UserDisplayName: Peter
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-09T18:08:28.533Z
Answers.Text: We want to estimate $\hat{g}$, which is an expectation over trajectories. Depending on the algorithm prior to PPO, this is done differently. It relates to the discussion of TD(0) vs n-step TD estimation (although the quantity in the expectation is different, the principle is similar).  What you propose in the comment is to take $n$ samples of 1-step tranistions, which is often done in off-policy methods (like DDPG). The other way, done in VPG is to collect one trajectory of length $T$. You can also take $n$ trajectories of length $T$ as in A3C. The A3C paper shows many ways to estimate it.
Answers.UserDisplayName: matorbi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-06T18:58:39.513Z
Answers.LastActivityDate: 2022-07-06T21:37:29.977Z
Answers.LastEditDate: 2022-07-06T21:37:29.977Z
Answers.UserDisplayName: matorbi
Body: <p>I understand how to derive the vanilla policy gradient
  <span class="math-container">$$
  \begin{align}
      \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t = 0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_{t} \mid s_{t}) \hat{A}^{\pi_{\theta}}(s_{t}, a_{t}) \right]
  \end{align}
  $$</span>
  as is also demonstrated in the <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html" rel="nofollow noreferrer">openai spinning up documentation</a>. Reading the <a href="https://arxiv.org/abs/1707.06347" rel="nofollow noreferrer">PPO paper</a>, they say that the most commonly used gradient estimator is
  <span class="math-container">$$
  \begin{align}
      \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[\nabla_{\theta}\log \pi_{\theta}(a_{t} \mid s_{t}) \hat{A}^{\pi_{\theta}}(s_{t}, a_{t}) \right]
  \end{align}
  $$</span>
  and from there they argue that this is equivalent to using the loss function
  <span class="math-container">$$
  \begin{align}
      \mathcal{L}^{\text{PG}}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[\log \pi_{\theta}(a_{t} \mid s_{t}) \hat{A}^{\pi_{\theta}}(s_{t}, a_{t}) \right]
  \end{align}
  $$</span>
  where the derivative is then taken with respect to <span class="math-container">$\theta$</span>.</p>
  <p><strong>Question:</strong> How does the vanilla policy gradient relate to the gradient stated in the PPO paper? More precisely, they seem to be identical up to the sum over <span class="math-container">$t$</span>. Why can the sum be ignored and the gradient is still the same?</p>
  <p>Thanks in advance for any help!</p>
  
Tags: <reinforcement-learning><deep-rl><policy-gradients><proximal-policy-optimization>
Title: Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?
UserDisplayName: Peter
ThreadText: # Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?
  
  (Asked by: username_0 on 2022-07-05)
  
  I understand how to derive the vanilla policy gradient
  $$
  \begin{align}
  \nabla\_{\theta}J(\pi\_{\theta}) = \mathbb{E}\_{\pi\_{\theta}} \left[ \sum\_{t = 0}^{T} \nabla\_{\theta} \log \pi\_{\theta}(a\_{t} \mid s\_{t}) \hat{A}^{\pi\_{\theta}}(s\_{t}, a\_{t}) \right]
  \end{align}
  $$
  as is also demonstrated in the [openai spinning up documentation](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html). Reading the [PPO paper](https://arxiv.org/abs/1707.06347), they say that the most commonly used gradient estimator is
  $$
  \begin{align}
  \nabla\_{\theta}J(\pi\_{\theta}) = \mathbb{E}\_{\pi\_{\theta}} \left[\nabla\_{\theta}\log \pi\_{\theta}(a\_{t} \mid s\_{t}) \hat{A}^{\pi\_{\theta}}(s\_{t}, a\_{t}) \right]
  \end{align}
  $$
  and from there they argue that this is equivalent to using the loss function
  $$
  \begin{align}
  \mathcal{L}^{\text{PG}}(\theta) = \mathbb{E}\_{\pi\_{\theta}} \left[\log \pi\_{\theta}(a\_{t} \mid s\_{t}) \hat{A}^{\pi\_{\theta}}(s\_{t}, a\_{t}) \right]
  \end{align}
  $$
  where the derivative is then taken with respect to $\theta$.
  
  **Question:** How does the vanilla policy gradient relate to the gradient stated in the PPO paper? More precisely, they seem to be identical up to the sum over $t$. Why can the sum be ignored and the gradient is still the same?
  
  Thanks in advance for any help!
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients, proximal-policy-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-06. Score: 3 | Accepted answer
  
  Short answer:
  
  The expectation $\mathbb{E}\_t$ in the PPO paper is not an expectation over trajectories, but a mean. I suppose the confusion comes from there. The two quantities are otherwise very similar. The gradient is however not the same, as it is a biased estimator.
  
  Long answer:
  
  There are many related formulations for RL problems, on one hand we have the finite-horizon (episodic) setting:
  
  $$ J\_1(\theta)=\mathbb{E}\_{\tau\sim p\_{\pi\_{\theta}}}\left[\sum\_{t= 0}^T r(s\_t,a\_t)\right]. $$
  
  On the other hand, we have the commonly used infinite-horizon setting (assumed by the PPO paper):
  
  $$ J\_2(\theta)=\mathbb{E}\_{\tau\sim p\_{\pi\_{\theta}}}\left[\sum\_{t\geq 0} \gamma^tr(s\_t,a\_t)\right]. $$
  
  In the finite horizon setting, we have :
  
  \begin{equation} \nabla\_{\theta} J\_1(\theta) = \mathbb{E}\_{\tau\sim p\_{\pi\_{\theta}}}\left[ \left(\sum\_{t= 0}^T \nabla\_{\theta}\log\pi\_{\theta}(a\_t\mid s\_t) \right)R(\tau) \right], \end{equation}
  
  where $R(\tau)=\sum\_{t= 0}^T r(s\_t,a\_t)$. As explained in the SpinningUp post, we can replace $R(\tau)$ by $A\_{\pi\_{\theta}}(s\_t,a\_t)$.
  
  Now, a similar proof shows that:
  
  \begin{equation} \nabla\_{\theta} J\_2(\theta) = \mathbb{E}\_{\tau\sim p\_{\pi\_{\theta}}}\left[ \sum\_{t\geq 0}\gamma^t \nabla\_{\theta}\log\pi\_{\theta}(a\_t\mid s\_t) A\_{\pi\_{\theta}}(s\_t,a\_t) \right]. \end{equation}
  
  Now, in practice it is very common to approximate
  
  $$\nabla\_{\theta} J\_2(\theta) \approx \mathbb{E}\_{\tau\sim p\_{\pi\_{\theta}}}\left[ \sum\_{t\geq 0}\nabla\_{\theta}\log\pi\_{\theta}(a\_t\mid s\_t) A\_{\pi\_{\theta}}(s\_t,a\_t) \right]:=g^{\gamma},$$
  
  that is we drop the discount factor $\gamma^t$. This relates to the policy gradient theorem and that we want to sample states from the steady-state distribution instead of the discounted state distribution.
  
  Now, we want to approximate $A\_{\pi}(s\_t,a\_t)$ by an estimator $\hat{A\_t}$. Schulman et al in the [GAE paper](https://arxiv.org/abs/1506.02438) introduce the definition of a $\gamma$-just estimator. The exact definition is maybe not essential, but $\gamma$-just estimators include $Q\_{\pi}(s\_t,a\_t)$, $A\_{\pi}(s\_t,a\_t)$ and the GAE estimator used in the PPO paper.
  
  So, if $\hat{A}\_t$ is $\gamma$-just, then we have:
  $$g^{\gamma} = \mathbb{E}\_{\tau\sim p\_{\pi\_{\theta}}}\left[ \sum\_{t\geq 0}\nabla\_{\theta}\log\pi\_{\theta}(a\_t\mid s\_t) \hat{A}\_t \right]. $$
  
  This expectation is however still not convenient for a practical algorithm, as we need to collect the trajectories with a policy with older weights $\pi\_{\theta\_{\text{old}}}$. So we approximate
  
  $$g^{\gamma} \approx \mathbb{E}\_{\tau\sim p\_{\pi\_{\theta\_{\text{old}}}}}\left[ \sum\_{t\geq 0}\nabla\_{\theta}\log\pi\_{\theta}(a\_t\mid s\_t) \hat{A}\_t \right]:=\hat{g}. $$
  
  Now, we need to approximate this expectation, so we collect $N$ rollouts of identical length $T$ (for simplicity) with the policy $\pi\_{\theta\_{\text{old}}}$, then we have an approximation of $g^{\gamma}$:
  
  $$\nabla\_{\theta} J\_2(\theta) \approx g^{\gamma}\approx \hat{g}\approx\frac{1}{N}\sum\_{n=0}^N\sum\_{t=0}^T \nabla\_{\theta}\log\pi\_{\theta}(a\_t^n\mid s\_t^n) \hat{A}\_t^n,$$
  which is what the $\mathbb{E}\_t$ in the PPO paper means. The expression on the right-hand side is now finally useful for gradient descent-like algorithms, as $\hat{A\_t}$does not depend on $\theta$.
  
  Note also that much of the PPO paper aims to ensure that the last approximation by replacing $p\_{\pi\_{\theta}}$ by $p\_{\pi\_{\theta\_{\text{old}}}}$ is valid.
  
  > **Comments:**
  > **username_0**: Thanks for the detailed explanation. Indeed, it does seem like the confusion stems from the $\hat{E}_{t}$ notation. I understand your derivation but I dont get why your last estimator $\hat{g}$ is what they refer to by $\hat{E}_{t}$. For me "an empirical batch over a finite batch of samples" is rather your last estimor without the sum over t, i.e. $\frac{1}{N} \sum_{n = 0}^{N} \nabla_{\theta} \log \pi_{\theta}(a_{t}^{n} \mid s_{t}^{n}) \hat{A}_{t}^{n}$.  
  > **username_1**: We want to estimate $\hat{g}$, which is an expectation over trajectories. Depending on the algorithm prior to PPO, this is done differently. It relates to the discussion of TD(0) vs n-step TD estimation (although the quantity in the expectation is different, the principle is similar).  What you propose in the comment is to take $n$ samples of 1-step tranistions, which is often done in off-policy methods (like DDPG). The other way, done in VPG is to collect one trajectory of length $T$. You can also take $n$ trajectories of length $T$ as in A3C. The A3C paper shows many ways to estimate it.  
  
  ---
  
  