Answers.Body: <p>Yes, the idea of PPO is to keep the updates small so that the new policy is not too far from the old policy. If you look at the left figure, this is the case as the absolute magnitude of L^clip is capped. The only region where this absolute magnitude is uncapped is on the right hand portion of the right figure. In this region, r is &gt; 1. Since r = new prob / old prob, it means the previous update has increased the probability of an action that resulted in a worse than expected outcome (hence the negative advantage). Therefore, we want to unroll that update and not capping the ratio will achieve that goal better.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-28T08:57:49.893Z
Answers.Text: Okay, so if I understand correctly, the fact that the right plot is not capped at 1+e, means that we can unroll that update at the next update step because then the gradient contribution at that specific time-step is non-zero. However, if it would be capped at 1+e, then the gradient would be zero, and unrolling would not occur. Do you agree?
Answers.UserDisplayName: Jer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-30T02:13:18.547Z
Answers.Text: Yes, that is my understanding. If it were capped at 1+e, unrolling should still occur if the ratio is below 1+e as you would still have non-zero gradients passing through. But if the ratio were large, which means the probability of those actions have changed a lot in the wrong direction, you end up not unrolling when it is most needed.
Answers.UserDisplayName: languageoftheuniverse
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-27T08:35:03.497Z
Answers.LastActivityDate: 2022-10-27T08:35:03.497Z
Answers.UserDisplayName: languageoftheuniverse
Answers.Body: <p>A positive advantage increases the probability of taking that action, hence <span class="math-container">$A_t &gt; 0$</span> means that the gradient update makes <span class="math-container">$r_t(\theta)$</span> larger. We don't want to take too big of a step, hence we only let <span class="math-container">$r_t(\theta)$</span> increase to <span class="math-container">$1 + \epsilon$</span> before we start ignoring that advantage.</p>
  <p>If <span class="math-container">$A_t &gt; 0$</span> but <span class="math-container">$r_t(\theta) &lt; 1 - \epsilon$</span> it must mean that there are many other gradient samples in the training batch that are pushing down <span class="math-container">$r_t(\theta)$</span>, because if we only had <span class="math-container">$A_t$</span> it would increase <span class="math-container">$r_t(\theta)$</span>. In this case you can see <span class="math-container">$A_t$</span> is actually pushing in the opposite direction of the gradient update. If <span class="math-container">$A_t &gt; 0$</span> and <span class="math-container">$r_t(\theta) &gt; 1 + \epsilon$</span> then <span class="math-container">$A_t$</span> is going in the same direction of the gradient update.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-26T22:09:23.237Z
Answers.LastActivityDate: 2022-11-26T22:09:23.237Z
Answers.UserDisplayName: Taw
Answers.Body: <p>There are six different situations:
  <a href="https://i.sstatic.net/BVJfI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/BVJfI.png" alt="enter image description here" /></a></p>
  <h2>Case 1 and 2: the ratio is between the range</h2>
  <p>In situations 1 and 2, <strong>the clipping does not apply since the ratio is between the range <span class="math-container">$[1 - \epsilon, 1 + \epsilon]$</span></strong></p>
  <p>In situation 1, we have a positive advantage: the <strong>action is better than the average</strong> of all the actions in that state. Therefore, we should encourage our current policy to increase the probability of taking that action in that state.</p>
  <p>Since the ratio is between intervals, <strong>we can increase our policy’s probability of taking that action at that state</strong>.</p>
  <p>In situation 2, we have a negative advantage: the action is worse than the average of all actions at that state. Therefore, we should discourage our current policy from taking that action in that state.</p>
  <p>Since the ratio is between intervals, ** the probability that our policy takes that action at that state**</p>
  <h2>Case 3 and 4: the ratio is below the range</h2>
  <p>If the probability ratio is lower than <span class="math-container">$[1 - \epsilon]$</span>, the probability of taking that action at that state is much lower than with the old policy.</p>
  <p>If, like in situation 3, the advantage estimate is positive (<span class="math-container">$A&gt;0$</span>), then <strong>you want to increase the probability of taking that action at that state</strong>.</p>
  <p>But if, like situation 4, the advantage estimate is negative, <strong>we don’t want to decrease further</strong> the probability of taking that action at that state. Therefore, the gradient is = 0 (since we’re on a flat line), so we don’t update our weights.</p>
  <h2>Case 5 and 6: the ratio is above the range</h2>
  <p>If the probability ratio is higher than <span class="math-container">$[1 + \epsilon]$</span>, the probability of taking that action at that state in the current policy is <strong>much higher than in the former policy</strong>.</p>
  <p>If, like in situation 5, the advantage is positive, <strong>we don’t want to get too greedy</strong>. We already have a higher probability of taking that action at that state than the former policy. Therefore, the gradient is = 0 (since we’re on a flat line), so we don’t update our weights.</p>
  <p>If, like in situation 6, the advantage is negative, we want to decrease the probability of taking that action at that state.</p>
  <p>So if we recap, <strong>we only update the policy with the unclipped objective part</strong>. When the minimum is the clipped objective part, we don’t update our policy weights since the gradient will equal 0.</p>
  <p>Source: <a href="https://huggingface.co/deep-rl-course/unit8/visualize" rel="nofollow noreferrer">DEEP RL Course - Visualize the Clipped Surrogate Objective Function</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-05T13:51:27.940Z
Answers.LastActivityDate: 2023-04-05T13:51:27.940Z
Answers.UserDisplayName: Deb
Answers.Body: <p>Quote from <a href="https://huggingface.co/learn/deep-rl-course/unit8/visualize" rel="nofollow noreferrer">https://huggingface.co/learn/deep-rl-course/unit8/visualize</a> :</p>
  <blockquote>
  <p>We update our policy only if:</p>
  <ul>
  <li>Our ratio is in the range</li>
  <li><strong>Our ratio is outside the range, but the advantage leads to getting    closer to the range</strong>
  <ul>
  <li>Being below the ratio but the advantage is &gt; 0</li>
  <li>Being above the ratio but the advantage is &lt; 0</li>
  </ul>
  </li>
  </ul>
  </blockquote>
  <p>And that's exactly what the image you've attached is showing. Keep updating while we are climbing towards the range. It's also fine to update within the Trust Region. Ignore if we overshoot this region and wanted to go even further.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-13T09:23:23.847Z
Answers.LastActivityDate: 2023-11-13T09:23:23.847Z
Answers.UserDisplayName: Serhiy
Body: <p>In PPO with clipped surrogate objective (see the paper <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="nofollow noreferrer">here</a>), we have the following objective:</p>
  <p><a href="https://i.sstatic.net/nDPGt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/nDPGt.png" alt="PPO objective" /></a></p>
  <p>The shape of the function is shown in the image below, and depends on whether the advantage is positive or negative.</p>
  <p><a href="https://i.sstatic.net/HtBV0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HtBV0.png" alt="objective plot over r" /></a></p>
  <p>The min() operator makes <span class="math-container">$L^{CLIP}(\theta)$</span> a lower bound to the original surrogate objective.
  But why do we want this lower bound? In other words, why clip only at <span class="math-container">$1+\epsilon$</span> when <span class="math-container">$A &gt; 0$</span> ?</p>
  <p>Isn't it important to keep the new policy in the neighborhood of the old policy, so that even <span class="math-container">$r_t(\theta) &lt; 1-\epsilon$</span> should be undesired?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-19T15:31:55.723Z
Comments.Text: In the PPO loss in PyTorch, they clip from both sides (i.e without considering the sign of Advantage), but it still works.
Comments.UserDisplayName: Wenuka
Tags: <reinforcement-learning><deep-rl><proximal-policy-optimization>
Title: Why clip the PPO objective on only one side?
UserDisplayName: Jer
ThreadText: # Why clip the PPO objective on only one side?
  
  (Asked by: username_0 on 2022-10-24)
  
  In PPO with clipped surrogate objective (see the paper [here](https://arxiv.org/pdf/1707.06347.pdf)), we have the following objective:
  
  [![PPO objective](https://i.sstatic.net/nDPGt.png)](https://i.sstatic.net/nDPGt.png)
  
  The shape of the function is shown in the image below, and depends on whether the advantage is positive or negative.
  
  [![objective plot over r](https://i.sstatic.net/HtBV0.png)](https://i.sstatic.net/HtBV0.png)
  
  The min() operator makes $L^{CLIP}(\theta)$ a lower bound to the original surrogate objective.
  But why do we want this lower bound? In other words, why clip only at $1+\epsilon$ when $A > 0$ ?
  
  Isn't it important to keep the new policy in the neighborhood of the old policy, so that even $r\_t(\theta) < 1-\epsilon$ should be undesired?
  
  **Tags:** reinforcement-learning, deep-rl, proximal-policy-optimization
  
  > **Comments:**
  > **username_1**: In the PPO loss in PyTorch, they clip from both sides (i.e without considering the sign of Advantage), but it still works.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_2 on 2022-10-27. Score: 1
  
  Yes, the idea of PPO is to keep the updates small so that the new policy is not too far from the old policy. If you look at the left figure, this is the case as the absolute magnitude of L^clip is capped. The only region where this absolute magnitude is uncapped is on the right hand portion of the right figure. In this region, r is > 1. Since r = new prob / old prob, it means the previous update has increased the probability of an action that resulted in a worse than expected outcome (hence the negative advantage). Therefore, we want to unroll that update and not capping the ratio will achieve that goal better.
  
  > **Comments:**
  > **username_0**: Okay, so if I understand correctly, the fact that the right plot is not capped at 1+e, means that we can unroll that update at the next update step because then the gradient contribution at that specific time-step is non-zero. However, if it would be capped at 1+e, then the gradient would be zero, and unrolling would not occur. Do you agree?  
  > **username_2**: Yes, that is my understanding. If it were capped at 1+e, unrolling should still occur if the ratio is below 1+e as you would still have non-zero gradients passing through. But if the ratio were large, which means the probability of those actions have changed a lot in the wrong direction, you end up not unrolling when it is most needed.  
  
  ---
  
  ### Answer by username_3 on 2022-11-26. Score: 1
  
  A positive advantage increases the probability of taking that action, hence $A\_t > 0$ means that the gradient update makes $r\_t(\theta)$ larger. We don't want to take too big of a step, hence we only let $r\_t(\theta)$ increase to $1 + \epsilon$ before we start ignoring that advantage.
  
  If $A\_t > 0$ but $r\_t(\theta) < 1 - \epsilon$ it must mean that there are many other gradient samples in the training batch that are pushing down $r\_t(\theta)$, because if we only had $A\_t$ it would increase $r\_t(\theta)$. In this case you can see $A\_t$ is actually pushing in the opposite direction of the gradient update. If $A\_t > 0$ and $r\_t(\theta) > 1 + \epsilon$ then $A\_t$ is going in the same direction of the gradient update.
  
  
  ---
  
  ### Answer by username_4 on 2023-04-05. Score: 0
  
  There are six different situations:
  [![enter image description here](https://i.sstatic.net/BVJfI.png)](https://i.sstatic.net/BVJfI.png)
  
  ## Case 1 and 2: the ratio is between the range
  
  In situations 1 and 2, **the clipping does not apply since the ratio is between the range $[1 - \epsilon, 1 + \epsilon]$**
  
  In situation 1, we have a positive advantage: the **action is better than the average** of all the actions in that state. Therefore, we should encourage our current policy to increase the probability of taking that action in that state.
  
  Since the ratio is between intervals, **we can increase our policy’s probability of taking that action at that state**.
  
  In situation 2, we have a negative advantage: the action is worse than the average of all actions at that state. Therefore, we should discourage our current policy from taking that action in that state.
  
  Since the ratio is between intervals, \*\* the probability that our policy takes that action at that state\*\*
  
  ## Case 3 and 4: the ratio is below the range
  
  If the probability ratio is lower than $[1 - \epsilon]$, the probability of taking that action at that state is much lower than with the old policy.
  
  If, like in situation 3, the advantage estimate is positive ($A>0$), then **you want to increase the probability of taking that action at that state**.
  
  But if, like situation 4, the advantage estimate is negative, **we don’t want to decrease further** the probability of taking that action at that state. Therefore, the gradient is = 0 (since we’re on a flat line), so we don’t update our weights.
  
  ## Case 5 and 6: the ratio is above the range
  
  If the probability ratio is higher than $[1 + \epsilon]$, the probability of taking that action at that state in the current policy is **much higher than in the former policy**.
  
  If, like in situation 5, the advantage is positive, **we don’t want to get too greedy**. We already have a higher probability of taking that action at that state than the former policy. Therefore, the gradient is = 0 (since we’re on a flat line), so we don’t update our weights.
  
  If, like in situation 6, the advantage is negative, we want to decrease the probability of taking that action at that state.
  
  So if we recap, **we only update the policy with the unclipped objective part**. When the minimum is the clipped objective part, we don’t update our policy weights since the gradient will equal 0.
  
  Source: [DEEP RL Course - Visualize the Clipped Surrogate Objective Function](https://huggingface.co/deep-rl-course/unit8/visualize)
  
  
  ---
  
  ### Answer by username_5 on 2023-11-13. Score: 0
  
  Quote from <https://huggingface.co/learn/deep-rl-course/unit8/visualize> :
  
  > We update our policy only if:
  >
  > * Our ratio is in the range
  > * **Our ratio is outside the range, but the advantage leads to getting closer to the range**
  >   + Being below the ratio but the advantage is > 0
  >   + Being above the ratio but the advantage is < 0
  
  And that's exactly what the image you've attached is showing. Keep updating while we are climbing towards the range. It's also fine to update within the Trust Region. Ignore if we overshoot this region and wanted to go even further.
  
  
  ---
  
  