Answers.Body: <p>Ok, so I think I have a better understanding of this now.</p>
  
  <p>Firstly, let's remind the main idea of the PPO : staying close to the previous policy. It's the same idea than in TRPO, but the L function is improved.</p>
  
  <p>So, you wanna make "small but safe steps". With clipped surrogate objective, you don't give too much importance to promising actions. You learn that bad actions are bad, so you decrease their probability according to "how bad" they are. But for good actions, you only learn that they are "a little bit good", and their probability will be just slightly increased.</p>
  
  <p>This mechanism allows you to perform small but relevant updates of your policy.</p>
  
  <p>hope this will help someone :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-07T09:09:57.587Z
Answers.Text: Yes I think you understood it well. See Figure 1 on page 3. The red dots depict "where the policy" currently is. If we move down along the line (reduction in objective), we do not do any clipping at all. If we move up along the line (improvement in objective), we also do actually allow that for a little bit, but start clipping if we move too far from the existing policy.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-07T08:59:57.400Z
Answers.LastActivityDate: 2018-09-07T08:59:57.400Z
Answers.UserDisplayName: 16Aghnar
Answers.Body: <p>I think <a href="https://ai.stackexchange.com/users/17759/16aghnar">@16Aghnar</a> explains the concept quite well. However, by clipping the surrogate objective alone doesn't ensure the trust region as stated in the paper:</p>
  
  <blockquote>
    <p><a href="https://openreview.net/attachment?id=r1etN1rtPB&amp;name=original_pdf" rel="nofollow noreferrer">Engstrom et al., 2020</a>, Implementation Matters in Deep RL: A Case Study on PPO and TRPO.</p>
  </blockquote>
  
  <p>The authors inspected OpenAI's implementation of PPO and find many code-level optimizations, I'll list the most important optimizations below:</p>
  
  <ol>
  <li>Clipped surrogate objective</li>
  <li>Value function clipping</li>
  <li>Reward scaling</li>
  <li>Orthogonal initialization and layer scaling</li>
  <li>Adam learning rate and annealing</li>
  </ol>
  
  <p>They find that: </p>
  
  <ul>
  <li><p>PPO-M (<code>2.</code>-<code>5.</code>) alone can maintain the trust region.</p>
  
  <p>(PPO-M: PPO without Clipped surrogate objective, but with code-level optimizations)</p></li>
  <li><p>PPO-Clip (<code>1.</code> only) cannot maintain the trust region.</p>
  
  <p>(PPO-Clip: PPO without code-level optimizations, but with Clipped surrogate objective)</p></li>
  <li><p>TRPO+ has better performance comparing to TRPO, and with similar performance comparing to PPO</p>
  
  <p>(TRPO+: TRPO with code-level optimizations used in PPO OpenAI implementation)</p></li>
  </ul>
  
  <p>A intuitive thought on why Clipped surrogate objective alone does not work is: The first step we take is unclipped.</p>
  
  <blockquote>
    <p>As a result, since we initialize <span class="math-container">$\pi_\theta$</span> as <span class="math-container">$\pi$</span> (and thus the ratios start all equal to one) the first step we take is identical to a maximization step over the unclipped surrogate reward. Therefore, the size of step we take is determined solely be the steepness of the surrogate landscape (i.e. Lipschitz constant of the  ptimization problem we solve), and we can end up moving arbitrarily far from the trust region. -- <a href="https://openreview.net/attachment?id=r1etN1rtPB&amp;name=original_pdf" rel="nofollow noreferrer">Engstrom et al., 2020</a></p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T16:31:07.403Z
Answers.LastActivityDate: 2020-04-14T16:31:07.403Z
Answers.UserDisplayName: J3soon
Body: <p>In <a href="https://arxiv.org/abs/1707.06347" rel="nofollow noreferrer">Proximal Policy Optimization Algorithms</a> (2017), Schulman et al. write</p>
  <blockquote>
  <p>With this scheme, we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse.</p>
  </blockquote>
  <p>I don't understand why the clipped surrogate objective works. How can it work if it doesn't take into account the objective improvements?</p>
  
Tags: <reinforcement-learning><papers><objective-functions><proximal-policy-optimization>
Title: Why does the clipped surrogate objective work in Proximal Policy Optimization?
UserDisplayName: 16Aghnar
ThreadText: # Why does the clipped surrogate objective work in Proximal Policy Optimization?
  
  (Asked by: username_0 on 2018-09-05)
  
  In [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347) (2017), Schulman et al. write
  
  > With this scheme, we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse.
  
  I don't understand why the clipped surrogate objective works. How can it work if it doesn't take into account the objective improvements?
  
  **Tags:** reinforcement-learning, papers, objective-functions, proximal-policy-optimization
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2018-09-07. Score: 2 | Accepted answer
  
  Ok, so I think I have a better understanding of this now.
  
  Firstly, let's remind the main idea of the PPO : staying close to the previous policy. It's the same idea than in TRPO, but the L function is improved.
  
  So, you wanna make "small but safe steps". With clipped surrogate objective, you don't give too much importance to promising actions. You learn that bad actions are bad, so you decrease their probability according to "how bad" they are. But for good actions, you only learn that they are "a little bit good", and their probability will be just slightly increased.
  
  This mechanism allows you to perform small but relevant updates of your policy.
  
  hope this will help someone :)
  
  > **Comments:**
  > **username_1**: Yes I think you understood it well. See Figure 1 on page 3. The red dots depict "where the policy" currently is. If we move down along the line (reduction in objective), we do not do any clipping at all. If we move up along the line (improvement in objective), we also do actually allow that for a little bit, but start clipping if we move too far from the existing policy.  
  
  ---
  
  ### Answer by username_2 on 2020-04-14. Score: 1
  
  I think [@username_0](https://ai.stackexchange.com/users/17759/username_0) explains the concept quite well. However, by clipping the surrogate objective alone doesn't ensure the trust region as stated in the paper:
  
  > [Engstrom et al., 2020](https://openreview.net/attachment?id=r1etN1rtPB&name=original_pdf), Implementation Matters in Deep RL: A Case Study on PPO and TRPO.
  
  The authors inspected OpenAI's implementation of PPO and find many code-level optimizations, I'll list the most important optimizations below:
  
  1. Clipped surrogate objective
  2. Value function clipping
  3. Reward scaling
  4. Orthogonal initialization and layer scaling
  5. Adam learning rate and annealing
  
  They find that:
  
  * PPO-M (`2.`-`5.`) alone can maintain the trust region.
  
    (PPO-M: PPO without Clipped surrogate objective, but with code-level optimizations)
  * PPO-Clip (`1.` only) cannot maintain the trust region.
  
    (PPO-Clip: PPO without code-level optimizations, but with Clipped surrogate objective)
  * TRPO+ has better performance comparing to TRPO, and with similar performance comparing to PPO
  
    (TRPO+: TRPO with code-level optimizations used in PPO OpenAI implementation)
  
  A intuitive thought on why Clipped surrogate objective alone does not work is: The first step we take is unclipped.
  
  > As a result, since we initialize $\pi\_\theta$ as $\pi$ (and thus the ratios start all equal to one) the first step we take is identical to a maximization step over the unclipped surrogate reward. Therefore, the size of step we take is determined solely be the steepness of the surrogate landscape (i.e. Lipschitz constant of the ptimization problem we solve), and we can end up moving arbitrarily far from the trust region. -- [Engstrom et al., 2020](https://openreview.net/attachment?id=r1etN1rtPB&name=original_pdf)
  
  
  ---
  
  