Answers.Body: <p>In my opinion both <em>&quot;Penalty&quot;</em> and <em>&quot;Constraint&quot;</em> have somewhat similar ideas to <em>&quot;KL-Cutoff&quot;</em>, but <em>&quot;KL-Stopping&quot;</em> is entirely different.</p>
  <p>The idea of policy regularization is to add some form of regularization to your loss function, much like adding L2-norm regularization in supervised learning. So for ppo you have:</p>
  <pre><code>ppo_loss = -torch.mean(torch.min(rho * adv, clip(rho, 1-eps, 1+eps) * adv))
  loss = ppo_loss + alpha * reg
  </code></pre>
  <p>Now, the first paper discusses different types of regularization, e.g. <strong>entropy regularization</strong> or <strong>KL-divergence regularization</strong>. And also explains how to go about choosing the parameter <em>alpha</em>:</p>
  <ul>
  <li>You could just treat it as a hyperparameter, just like you would treat the learning rate. (Penalty)</li>
  <li>Or, you could optimize this parameter using a more sophisticated procedure. (Constraint)</li>
  </ul>
  <p><strong>You can add these regularization techniques to any policy gradient algorithm</strong>, e.g. vanilla policy gradient, actor-critic, or ppo.</p>
  <p>KL-Cutoff and KL-Stopping from the second paper actually address a different problem. Using the ppo objective you are allowed to perform many policy updates before throwing away your data and performing new rollouts. But if you update <em>too many</em> times then your training might be unstable. So after every update you check the KL-divergence between the old policy (behavioral policy) and the current policy. If the KL-divergence reaches a given threshold, then you should do something:</p>
  <ul>
  <li>KL-Stopping says to stop updating your policy, throw away the data, and rollout the new policy to collect new data.</li>
  <li>KL-Cutoff suggests something very similar to the idea of policy regularization: continue updating your policy, but add the term <span class="math-container">$\pm \alpha(D_{KL} - D_{thr})^2$</span> to the loss function. As far as I understand the authors suggest a fixed parameter alpha, which is the same as &quot;Penalty&quot; from the fist paper. However, your regularizer is <strong>not the KL-divergence</strong>, but how much you are violating the threshold.</li>
  </ul>
  <p>In my opinion these techniques are complementary. If you want to see this implemented in code I will add here a link to one of my repos where I implemented ppo:</p>
  <ul>
  <li><a href="https://github.com/pi-tau/playing-with-RL-models/blob/a411ff02cace759428af212a0f0086fd6b0fcf87/src/ppo.py#L202" rel="nofollow noreferrer">here</a> I use policy regularization with entropy,</li>
  <li><a href="https://github.com/pi-tau/playing-with-RL-models/blob/a411ff02cace759428af212a0f0086fd6b0fcf87/src/ppo.py#L150" rel="nofollow noreferrer">here</a> I use KL-stopping.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-19T14:52:20.343Z
Answers.LastActivityDate: 2023-09-20T11:07:20.517Z
Answers.LastEditDate: 2023-09-20T11:07:20.517Z
Answers.UserDisplayName: pi-tau
Body: <p>I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the following terms are equivalent -</p>
  <p><a href="https://arxiv.org/pdf/2006.05990.pdf" rel="nofollow noreferrer">This</a> paper talks about Policy regularization using KL Divergence
  <a href="https://i.sstatic.net/iu1q8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/iu1q8.png" alt="enter image description here" /></a></p>
  <p>Whereas <a href="https://arxiv.org/pdf/2010.13083.pdf" rel="nofollow noreferrer">this</a> paper uses the terms KL Stopping and KL Cutoff -</p>
  <p><a href="https://i.sstatic.net/U2W9Z.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/U2W9Z.png" alt="enter image description here" /></a></p>
  <p>I think &quot;Penalty&quot; from the first paper is the same as &quot;KL-cutoff&quot;. Also &quot;Constraint&quot; from the first paper is the same as &quot;KL-Stopping&quot;. Could someone let me know if I am correct?</p>
  
Tags: <reinforcement-learning><deep-rl><proximal-policy-optimization><on-policy-methods>
Title: Understanding KL Stopping and KL Cutoff for the PPO algorithm
UserDisplayName: desert_ranger
ThreadText: # Understanding KL Stopping and KL Cutoff for the PPO algorithm
  
  (Asked by: username_0 on 2023-09-19)
  
  I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the following terms are equivalent -
  
  [This](https://arxiv.org/pdf/2006.05990.pdf) paper talks about Policy regularization using KL Divergence
  [![enter image description here](https://i.sstatic.net/iu1q8.png)](https://i.sstatic.net/iu1q8.png)
  
  Whereas [this](https://arxiv.org/pdf/2010.13083.pdf) paper uses the terms KL Stopping and KL Cutoff -
  
  [![enter image description here](https://i.sstatic.net/U2W9Z.png)](https://i.sstatic.net/U2W9Z.png)
  
  I think "Penalty" from the first paper is the same as "KL-cutoff". Also "Constraint" from the first paper is the same as "KL-Stopping". Could someone let me know if I am correct?
  
  **Tags:** reinforcement-learning, deep-rl, proximal-policy-optimization, on-policy-methods
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-19. Score: 2 | Accepted answer
  
  In my opinion both "Penalty" and "Constraint" have somewhat similar ideas to "KL-Cutoff", but "KL-Stopping" is entirely different.
  The idea of policy regularization is to add some form of regularization to your loss function, much like adding L2-norm regularization in supervised learning. So for ppo you have:
  ppo_loss = -torch.mean(torch.min(rho * adv, clip(rho, 1-eps, 1+eps) * adv))
  loss = ppo_loss + alpha * reg
  
  Now, the first paper discusses different types of regularization, e.g. entropy regularization or KL-divergence regularization. And also explains how to go about choosing the parameter alpha:
  
  You could just treat it as a hyperparameter, just like you would treat the learning rate. (Penalty)
  Or, you could optimize this parameter using a more sophisticated procedure. (Constraint)
  
  You can add these regularization techniques to any policy gradient algorithm, e.g. vanilla policy gradient, actor-critic, or ppo.
  KL-Cutoff and KL-Stopping from the second paper actually address a different problem. Using the ppo objective you are allowed to perform many policy updates before throwing away your data and performing new rollouts. But if you update too many times then your training might be unstable. So after every update you check the KL-divergence between the old policy (behavioral policy) and the current policy. If the KL-divergence reaches a given threshold, then you should do something:
  
  KL-Stopping says to stop updating your policy, throw away the data, and rollout the new policy to collect new data.
  KL-Cutoff suggests something very similar to the idea of policy regularization: continue updating your policy, but add the term $\pm \alpha(D_{KL} - D_{thr})^2$ to the loss function. As far as I understand the authors suggest a fixed parameter alpha, which is the same as "Penalty" from the fist paper. However, your regularizer is not the KL-divergence, but how much you are violating the threshold.
  
  In my opinion these techniques are complementary. If you want to see this implemented in code I will add here a link to one of my repos where I implemented ppo:
  
  here I use policy regularization with entropy,
  here I use KL-stopping.
  
  
  ---
  
  