Answers.Body: <p>No approximation here. The formula for <span class="math-container">$L^{CPI}$</span> can be directly derived from the formula for <span class="math-container">$L^{PG}$</span>.</p>
  <p><span class="math-container">\begin{align}
  \nabla L^{PG} &amp;= \mathbb{E}_{a_t, s_t \sim \pi_\theta} \Big[ \nabla \log \pi_\theta(a_t | s_t) A_t \Big] \\
  &amp;= \int_{a, s} \pi_\theta(a | s) \nabla \log \pi_\theta(a|s) A_t \\
  &amp;=  \int_{a, s}  \frac{\pi_{\theta_{old}}(a | s)}{\pi_{\theta_{old}}(a | s)} \pi_\theta(a | s) \frac{\nabla \pi_\theta(a | s)}{\pi_\theta(a | s)} A_t \\
  &amp;= \mathbb{E}_{a_t, s_t \sim \pi_{\theta_{old}}} \Big[ \frac{\nabla \pi_\theta(a_s | s_t)}{\pi_{\theta_{old}} (a_t | s_t)} A_t \Big] \\
  &amp;= \nabla L^{CPI}
  \end{align}</span></p>
  <p>Now:
  <span class="math-container">\begin{align}
  L^{PG} &amp;= \int \nabla L^{PG} = \mathbb{E}_{a_t, s_t \sim \pi_\theta} \Big[ \log \pi_\theta(a_t | s_t) A_t \Big] \\
  L^{CPI} &amp;= \int \nabla L^{CPI} = \mathbb{E}_{a_t, s_t \sim \pi_{\theta_{old}}} \Big[ \frac{\pi_\theta(a_s | s_t)}{\pi_{\theta_{old}} (a_t | s_t)} A_t \Big]
  \end{align}</span></p>
  <p>The difference between the two is that in <span class="math-container">$L^{PG}$</span> states and actions are sampled using the original policy <span class="math-container">$\pi_\theta$</span>. This means that once you perform one gradient update step you have to throw away the data and collect new.<br />
  In <span class="math-container">$L^{CPI}$</span> the states and actions are collected under <span class="math-container">$\pi_{\theta_{old}}$</span>, which means that you can perform multiple updates with the same dataset. Obviously, if you perform only one update step with PPO you will get the vanilla PG. After the first update step the two objectives will differ.</p>
  <p>There is a bit more theory regarding why you want to clip, but since this is not part of the question I will just refer you to this blog post that I wrote:<br />
  <a href="https://pi-tau.github.io/posts/actor-critic/#ppo" rel="nofollow noreferrer">https://pi-tau.github.io/posts/actor-critic/#ppo</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-12T15:53:01.633Z
Answers.Text: Thank you for the answer! I was just wondering - I was checking the CPI paper but I dont see where they propose it as an objective? Any idea?
Answers.UserDisplayName: craaaft
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-26T12:52:23.557Z
Answers.LastActivityDate: 2023-09-25T15:44:08.187Z
Answers.LastEditDate: 2023-09-25T15:44:08.187Z
Answers.UserDisplayName: pi-tau
Answers.Body: <p>The ratio term appeared between Eq. (13)&amp;(14) in <a href="https://arxiv.org/pdf/1502.05477.pdf" rel="nofollow noreferrer">Sch+15b</a>, as a result of importance sampling:</p>
  <p><span class="math-container">$$
  L^{\theta^{\prime}}(\theta)=E_{\left(s_t, a_t\right) \sim \pi_{\theta^{\prime}}}\left[\frac{\pi_\theta\left(a_t | s_t\right)}{\pi_{\theta^{\prime}}\left(a_t | s_t\right)} A^{\theta^{\prime}}\left(s_t, a_t\right)\right].
  $$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-11T08:13:42.367Z
Answers.LastActivityDate: 2023-05-11T08:13:42.367Z
Answers.UserDisplayName: Arya
Body: <p>I will use the notation used in the <a href="https://arxiv.org/abs/1707.06347" rel="nofollow noreferrer">proximal policy optimization paper</a>.</p>
  <p><a href="https://i.sstatic.net/uxxJk.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uxxJk.png" alt="conservative policy iteration objective" /></a></p>
  <p>What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio <span class="math-container">$r_t(\theta)$</span>?</p>
  <p>Put another way, we start with the vanilla policy gradient objective and aim to optimize <span class="math-container">$L^\text{PG} (\theta) = \mathbb{E}_t \left [  \log \pi_\theta(a_t|s_t)A_t \right]$</span>. Where in <span class="math-container">$L^\text{PG}(\theta)$</span> do I make an approximation to derive equation (6)?</p>
  <p>I could not find this objective following the reference <a href="https://people.eecs.berkeley.edu/%7Epabbeel/cs287-fa09/readings/KakadeLangford-icml2002.pdf" rel="nofollow noreferrer">KL02</a>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-11T07:13:38.070Z
Comments.Text: I am not sure either. It doesn’t make sense. Im confused because its not symmetric(unclipped). If the advantage is positive and the ratio is greater then 1, you get a big positive gradient. But if you have a negative advantage and a ratio less then 1(same scenario but flipped in my mind), then you get a small negative gradient. In my mind, given the same advantage magnitude, opposite signs, the gradient of a ratio of 2 would be the negative of a gradient of a ratio of 0.5. Does that make sense?
Comments.UserDisplayName: Yuh
Tags: <reinforcement-learning><papers><math><policy-gradients><proximal-policy-optimization>
Title: Where does the proximal policy optimization objective's ratio term come from?
UserDisplayName: fool
ThreadText: # Where does the proximal policy optimization objective's ratio term come from?
  
  (Asked by: username_0 on 2022-11-20)
  
  I will use the notation used in the [proximal policy optimization paper](https://arxiv.org/abs/1707.06347).
  
  [![conservative policy iteration objective](https://i.sstatic.net/uxxJk.png)](https://i.sstatic.net/uxxJk.png)
  
  What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r\_t(\theta)$?
  
  Put another way, we start with the vanilla policy gradient objective and aim to optimize $L^\text{PG} (\theta) = \mathbb{E}\_t \left [ \log \pi\_\theta(a\_t|s\_t)A\_t \right]$. Where in $L^\text{PG}(\theta)$ do I make an approximation to derive equation (6)?
  
  I could not find this objective following the reference [KL02](https://people.eecs.berkeley.edu/%7Epabbeel/cs287-fa09/readings/KakadeLangford-icml2002.pdf).
  
  **Tags:** reinforcement-learning, papers, math, policy-gradients, proximal-policy-optimization
  
  > **Comments:**
  > **username_1**: I am not sure either. It doesn’t make sense. Im confused because its not symmetric(unclipped). If the advantage is positive and the ratio is greater then 1, you get a big positive gradient. But if you have a negative advantage and a ratio less then 1(same scenario but flipped in my mind), then you get a small negative gradient. In my mind, given the same advantage magnitude, opposite signs, the gradient of a ratio of 2 would be the negative of a gradient of a ratio of 0.5. Does that make sense?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-08-26. Score: 1
  
  No approximation here. The formula for $L^{CPI}$ can be directly derived from the formula for $L^{PG}$.
  
  \begin{align}
  \nabla L^{PG} &= \mathbb{E}\_{a\_t, s\_t \sim \pi\_\theta} \Big[ \nabla \log \pi\_\theta(a\_t | s\_t) A\_t \Big] \\
  &= \int\_{a, s} \pi\_\theta(a | s) \nabla \log \pi\_\theta(a|s) A\_t \\
  &= \int\_{a, s} \frac{\pi\_{\theta\_{old}}(a | s)}{\pi\_{\theta\_{old}}(a | s)} \pi\_\theta(a | s) \frac{\nabla \pi\_\theta(a | s)}{\pi\_\theta(a | s)} A\_t \\
  &= \mathbb{E}\_{a\_t, s\_t \sim \pi\_{\theta\_{old}}} \Big[ \frac{\nabla \pi\_\theta(a\_s | s\_t)}{\pi\_{\theta\_{old}} (a\_t | s\_t)} A\_t \Big] \\
  &= \nabla L^{CPI}
  \end{align}
  
  Now:
  \begin{align}
  L^{PG} &= \int \nabla L^{PG} = \mathbb{E}\_{a\_t, s\_t \sim \pi\_\theta} \Big[ \log \pi\_\theta(a\_t | s\_t) A\_t \Big] \\
  L^{CPI} &= \int \nabla L^{CPI} = \mathbb{E}\_{a\_t, s\_t \sim \pi\_{\theta\_{old}}} \Big[ \frac{\pi\_\theta(a\_s | s\_t)}{\pi\_{\theta\_{old}} (a\_t | s\_t)} A\_t \Big]
  \end{align}
  
  The difference between the two is that in $L^{PG}$ states and actions are sampled using the original policy $\pi\_\theta$. This means that once you perform one gradient update step you have to throw away the data and collect new.  
  In $L^{CPI}$ the states and actions are collected under $\pi\_{\theta\_{old}}$, which means that you can perform multiple updates with the same dataset. Obviously, if you perform only one update step with PPO you will get the vanilla PG. After the first update step the two objectives will differ.
  
  There is a bit more theory regarding why you want to clip, but since this is not part of the question I will just refer you to this blog post that I wrote:  
  <https://username_2.github.io/posts/actor-critic/#ppo>
  
  > **Comments:**
  > **username_3**: Thank you for the answer! I was just wondering - I was checking the CPI paper but I dont see where they propose it as an objective? Any idea?  
  
  ---
  
  ### Answer by username_4 on 2023-05-11. Score: 0
  
  The ratio term appeared between Eq. (13)&(14) in [Sch+15b](https://arxiv.org/pdf/1502.05477.pdf), as a result of importance sampling:
  
  $$
  L^{\theta^{\prime}}(\theta)=E\_{\left(s\_t, a\_t\right) \sim \pi\_{\theta^{\prime}}}\left[\frac{\pi\_\theta\left(a\_t | s\_t\right)}{\pi\_{\theta^{\prime}}\left(a\_t | s\_t\right)} A^{\theta^{\prime}}\left(s\_t, a\_t\right)\right].
  $$
  
  
  ---
  
  