Answers.Body: <p>The loss in PPO has a penalty</p>
  <p><span class="math-container">$$-\text{kl_coeff_val}\cdot\mathrm{KL}(\pi^{\text{NN}}||\pi^{\text{Target}}).$$</span></p>
  <p>Increasing this coefficient means a lower reward, so it incentivizes a smaller KL divergence. Likewise, decreasing it allows a higher KL divergence.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-04T19:10:02.580Z
Answers.LastActivityDate: 2024-05-09T14:01:18.387Z
Answers.LastEditDate: 2024-05-09T14:01:18.387Z
Answers.UserDisplayName: Zhenghao Peng
Answers.Body: <p>Please note that the <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="nofollow noreferrer">official paper</a> introduces <strong>two</strong> different algorithms for proximal policy optimization: PPO-CLIP (Sec. 3) and PPO-Penalty (Sec. 4)</p>
  <p>The algorithm that you are describing is PPO-Penalty and the objective function is given by the following equation:</p>
  <p><span class="math-container">$$ L^{KL} = \mathbb{E} \bigg[ \frac{\pi_\theta (a|s)}{\pi_{\theta_{old}} (a|s)} A(a,s) - \beta \text{KL}(\pi_\theta, \pi_{\theta_{old}}) \bigg]$$</span></p>
  <p>Here <span class="math-container">$\beta$</span> is <strong>not</strong> a fixed hyperparameter, but is a Lagrangian multiplier for the constraint. In this case the constraint is that <span class="math-container">$\text{KL}(\pi_\theta, \pi_{\theta_{old}}) \leq KL_{tgt}$</span>.</p>
  <p>During optimization you want to increase the multiplier if the constraint is violated, and lower it otherwise.</p>
  <p><em><strong>Why do we want to do that?</strong></em></p>
  <p>If your constraint is violated too much, you want to increase the multiplier <span class="math-container">$\beta$</span> so that the next update step will pay more attention to this term and will try to take smaller update steps on the policy network. If the constraint is not violated, then you decrease the multiplier <span class="math-container">$\beta$</span> so that we can take larger update steps on the policy network.</p>
  <p>In theory you should converge to an optimal coefficient <span class="math-container">$\beta$</span> such that you take the largest update step on the policy network, such that the constraint is not violated. In this case the constraint will be fulfilled exactly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-28T09:01:36.917Z
Answers.LastActivityDate: 2023-07-28T09:01:36.917Z
Answers.UserDisplayName: pi-tau
Body: <p>I am using RLlib (Ray 1.4.0)'s implementation of PPO for a multi-agent scenario with continuous actions, and I find that the loss includes the KL divergence penalty term, apart from the surrogate loss, value function loss, and entropy.</p>
  <p>The KL coefficient is updated in the update_kl() function as follows:</p>
  <pre><code>    if sampled_kl &gt; 2.0 * self.kl_target:
          self.kl_coeff_val *= 1.5
      # Decrease.
      elif sampled_kl &lt; 0.5 * self.kl_target:
          self.kl_coeff_val *= 0.5
      # No change.
      else:
          return self.kl_coeff_val
  </code></pre>
  <p>I don't understand the reasoning behind this. If the point of the KL &quot;target&quot; is to reach the target, then why do the conditions above imply that the KL coefficient is getting larger (<strong>multiplied by 1.5 when the sampled KL is already found to be larger than the target?</strong>) when it is supposed to be made smaller instead? I feel like I am missing something here, but I am not able to get my head around it.</p>
  <p>I would appreciate any insights on this. Thank you.</p>
  
Tags: <reinforcement-learning><proximal-policy-optimization><loss><continuous-action-spaces><kl-divergence>
Title: KL divergence coefficient update doesn't make sense in RLlib's PPO implementation
UserDisplayName: hridayns
ThreadText: # KL divergence coefficient update doesn't make sense in RLlib's PPO implementation
  
  (Asked by: username_0 on 2021-07-18)
  
  I am using RLlib (Ray 1.4.0)'s implementation of PPO for a multi-agent scenario with continuous actions, and I find that the loss includes the KL divergence penalty term, apart from the surrogate loss, value function loss, and entropy.
  
  The KL coefficient is updated in the update\_kl() function as follows:
  
  ```
      if sampled_kl > 2.0 * self.kl_target:
          self.kl_coeff_val *= 1.5
      # Decrease.
      elif sampled_kl < 0.5 * self.kl_target:
          self.kl_coeff_val *= 0.5
      # No change.
      else:
          return self.kl_coeff_val
  
  ```
  
  I don't understand the reasoning behind this. If the point of the KL "target" is to reach the target, then why do the conditions above imply that the KL coefficient is getting larger (**multiplied by 1.5 when the sampled KL is already found to be larger than the target?**) when it is supposed to be made smaller instead? I feel like I am missing something here, but I am not able to get my head around it.
  
  I would appreciate any insights on this. Thank you.
  
  **Tags:** reinforcement-learning, proximal-policy-optimization, loss, continuous-action-spaces, kl-divergence
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-11-04. Score: 1
  
  The loss in PPO has a penalty
  
  $$-\text{kl\_coeff\_val}\cdot\mathrm{KL}(\pi^{\text{NN}}||\pi^{\text{Target}}).$$
  
  Increasing this coefficient means a lower reward, so it incentivizes a smaller KL divergence. Likewise, decreasing it allows a higher KL divergence.
  
  
  ---
  
  ### Answer by username_2 on 2023-07-28. Score: -1
  
  Please note that the [official paper](https://arxiv.org/pdf/1707.06347.pdf) introduces **two** different algorithms for proximal policy optimization: PPO-CLIP (Sec. 3) and PPO-Penalty (Sec. 4)
  
  The algorithm that you are describing is PPO-Penalty and the objective function is given by the following equation:
  
  $$ L^{KL} = \mathbb{E} \bigg[ \frac{\pi\_\theta (a|s)}{\pi\_{\theta\_{old}} (a|s)} A(a,s) - \beta \text{KL}(\pi\_\theta, \pi\_{\theta\_{old}}) \bigg]$$
  
  Here $\beta$ is **not** a fixed hyperparameter, but is a Lagrangian multiplier for the constraint. In this case the constraint is that $\text{KL}(\pi\_\theta, \pi\_{\theta\_{old}}) \leq KL\_{tgt}$.
  
  During optimization you want to increase the multiplier if the constraint is violated, and lower it otherwise.
  
  ***Why do we want to do that?***
  
  If your constraint is violated too much, you want to increase the multiplier $\beta$ so that the next update step will pay more attention to this term and will try to take smaller update steps on the policy network. If the constraint is not violated, then you decrease the multiplier $\beta$ so that we can take larger update steps on the policy network.
  
  In theory you should converge to an optimal coefficient $\beta$ such that you take the largest update step on the policy network, such that the constraint is not violated. In this case the constraint will be fulfilled exactly.
  
  
  ---
  
  