Body: <p>After some amount of training on a custom Multi-agent sparse-reward environment using RLlib's (1.4.0) PPO network, I found that my continuous actions turn into nan (explodes?) which is probably caused by a bad gradient update which in turn depends on the loss/objective function.</p>
  <p>As I understand it, PPO's loss function relies on three terms:</p>
  <ol>
  <li><p>The <strong>clipped surrogate objective</strong> which depends on outputs of old policy and new policy, the advantage, and the &quot;clip&quot; parameter(=0.3)</p>
  </li>
  <li><p>The <strong>Value Function Loss</strong></p>
  </li>
  <li><p>The <strong>Entropy Loss</strong> [mainly there to encourage exploration]</p>
  </li>
  </ol>
  <p><strong>Total Loss</strong> = <strong>Surrogate objective (clipped)</strong> - vf_loss_coeff * <strong>VF Loss</strong> + entropy_coeff * <strong>entropy</strong>.</p>
  <p><a href="https://i.sstatic.net/RxmqJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RxmqJ.png" alt="enter image description here" /></a></p>
  <p>The surrogate loss
  ( Reference: <a href="https://arxiv.org/abs/1707.06347" rel="nofollow noreferrer">https://arxiv.org/abs/1707.06347</a> )</p>
  <p>I have a bunch of questions:</p>
  <ol>
  <li><p>Is the <strong>ratio r<sub>t</sub>(theta)</strong> used that of the actual actions taken from new policy vs. old policy or is it the <strong>probability distributions</strong> of those actions? (since actions are continuous?)</p>
  </li>
  <li><p>Follow up question to 1: Assuming it is probability, <strong>can the probability ever be 0?</strong> Because if it is ever 0, then log probability would result in log(0) = inf/undefined - which would prove that is the root cause?</p>
  </li>
  <li><p>If 1 and 2 are safely debunked, then do I</p>
  </li>
  </ol>
  <p><strong>(A)</strong> lower my learning rate?</p>
  <p><strong>(B)</strong> Reduce the number of layers in my network?</p>
  <p><strong>(C)</strong> Use gradient clipping or action or reward clipping of some sort?</p>
  <p>To anyone who would be kind enough to share any insights into the matter, you have my gratitude.</p>
  <p>For more information, see relevant part of <strong>progress table below</strong>: where the <strong>total loss becomes inf</strong>. The only change I found is that the policy loss was all negative until row <strong>#445</strong>.</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th></th>
  <th>Total loss</th>
  <th>policy loss</th>
  <th>VF loss</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>430</td>
  <td>6.068537</td>
  <td>-0.053691725999999995</td>
  <td>6.102932</td>
  </tr>
  <tr>
  <td>431</td>
  <td>5.9919114</td>
  <td>-0.046943977000000005</td>
  <td>6.0161843</td>
  </tr>
  <tr>
  <td>432</td>
  <td>8.134636</td>
  <td>-0.05247503</td>
  <td>8.164852</td>
  </tr>
  <tr>
  <td>433</td>
  <td>4.222730599999999</td>
  <td>-0.048518334</td>
  <td>4.2523246</td>
  </tr>
  <tr>
  <td>434</td>
  <td>6.563492</td>
  <td>-0.05237444</td>
  <td>6.594456</td>
  </tr>
  <tr>
  <td>435</td>
  <td>8.171028999999999</td>
  <td>-0.048245672</td>
  <td>8.198222999999999</td>
  </tr>
  <tr>
  <td>436</td>
  <td>8.948264</td>
  <td>-0.048484523</td>
  <td>8.976327000000001</td>
  </tr>
  <tr>
  <td>437</td>
  <td>7.556602000000001</td>
  <td>-0.054372005</td>
  <td>7.5880575</td>
  </tr>
  <tr>
  <td>438</td>
  <td>6.124418</td>
  <td>-0.05249534</td>
  <td>6.155608999999999</td>
  </tr>
  <tr>
  <td>439</td>
  <td>4.267647</td>
  <td>-0.052565258</td>
  <td>4.2978816</td>
  </tr>
  <tr>
  <td>440</td>
  <td>4.912957700000001</td>
  <td>-0.054498855</td>
  <td>4.9448576</td>
  </tr>
  <tr>
  <td>441</td>
  <td>16.630292999999998</td>
  <td>-0.043477765999999994</td>
  <td>16.656229</td>
  </tr>
  <tr>
  <td>442</td>
  <td>6.3149705</td>
  <td>-0.057527818</td>
  <td>6.349851999999999</td>
  </tr>
  <tr>
  <td>443</td>
  <td>4.2269225</td>
  <td>-0.05446908599999999</td>
  <td>4.260793700000001</td>
  </tr>
  <tr>
  <td>444</td>
  <td>9.503102</td>
  <td>-0.052135203</td>
  <td>9.53277</td>
  </tr>
  <tr>
  <td>445</td>
  <td><strong>inf</strong></td>
  <td><strong>0.2436709</strong></td>
  <td><strong>4.410831</strong></td>
  </tr>
  <tr>
  <td>446</td>
  <td>nan</td>
  <td>-0.00029848056</td>
  <td>22.596403</td>
  </tr>
  <tr>
  <td>447</td>
  <td>nan</td>
  <td>0.00013323531</td>
  <td>0.00043436907999999994</td>
  </tr>
  <tr>
  <td>448</td>
  <td>nan</td>
  <td>1.5656527000000002e-05</td>
  <td>0.0002645221</td>
  </tr>
  <tr>
  <td>449</td>
  <td>nan</td>
  <td>1.3344318000000001e-05</td>
  <td>0.0003139485</td>
  </tr>
  <tr>
  <td>450</td>
  <td>nan</td>
  <td>6.941916999999999e-05</td>
  <td>0.00025863337</td>
  </tr>
  <tr>
  <td>451</td>
  <td>nan</td>
  <td>0.00015686743</td>
  <td>0.00013607396</td>
  </tr>
  <tr>
  <td>452</td>
  <td>nan</td>
  <td>-5.0206604e-06</td>
  <td>0.00027541115000000003</td>
  </tr>
  <tr>
  <td>453</td>
  <td>nan</td>
  <td>-4.5543664e-05</td>
  <td>0.0004247162</td>
  </tr>
  <tr>
  <td>454</td>
  <td>nan</td>
  <td>8.841756999999999e-05</td>
  <td>0.00020278389999999998</td>
  </tr>
  <tr>
  <td>455</td>
  <td>nan</td>
  <td>-8.465959e-05</td>
  <td>9.261127e-05</td>
  </tr>
  <tr>
  <td>456</td>
  <td>nan</td>
  <td>3.8680790000000003e-05</td>
  <td>0.00032097592999999995</td>
  </tr>
  <tr>
  <td>457</td>
  <td>nan</td>
  <td>2.7373152999999996e-06</td>
  <td>0.0005146417</td>
  </tr>
  <tr>
  <td>458</td>
  <td>nan</td>
  <td>-6.271608e-06</td>
  <td>0.0013273798000000001</td>
  </tr>
  <tr>
  <td>459</td>
  <td>nan</td>
  <td>-0.00013192794</td>
  <td>0.00030621013</td>
  </tr>
  <tr>
  <td>460</td>
  <td>nan</td>
  <td>0.00038987884</td>
  <td>0.00038019830000000004</td>
  </tr>
  <tr>
  <td>461</td>
  <td>nan</td>
  <td>-3.2747877999999998e-06</td>
  <td>0.00031471922</td>
  </tr>
  <tr>
  <td>462</td>
  <td>nan</td>
  <td>-6.9349815e-05</td>
  <td>0.00038836736000000006</td>
  </tr>
  <tr>
  <td>463</td>
  <td>nan</td>
  <td>-4.666238e-05</td>
  <td>0.0002851575</td>
  </tr>
  <tr>
  <td>464</td>
  <td>nan</td>
  <td>-3.7067155e-05</td>
  <td>0.00020161088</td>
  </tr>
  <tr>
  <td>465</td>
  <td>nan</td>
  <td>3.0623291e-06</td>
  <td>0.00019258813999999998</td>
  </tr>
  <tr>
  <td>466</td>
  <td>nan</td>
  <td>-8.599938e-06</td>
  <td>0.00036465342000000005</td>
  </tr>
  <tr>
  <td>467</td>
  <td>nan</td>
  <td>-1.1529375e-05</td>
  <td>0.00016500981</td>
  </tr>
  <tr>
  <td>468</td>
  <td>nan</td>
  <td>-3.0851965e-07</td>
  <td>0.00022042097</td>
  </tr>
  <tr>
  <td>469</td>
  <td>nan</td>
  <td>-0.0001133984</td>
  <td>0.00030230957999999997</td>
  </tr>
  <tr>
  <td>470</td>
  <td>nan</td>
  <td>-1.0735256e-05</td>
  <td>0.00034000343000000003</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p><strong>Optional</strong></p>
  <p>For even further context, check my related <a href="https://stackoverflow.com/questions/68249391/rllib-ppo-continuous-actions-seem-to-become-nan-after-total-loss-inf">question</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-05T14:16:26.453Z
Comments.Text: Hello. Please, do not just link us to another external post/article to provide/describe the context. Make sure you include in this post all the information needed to answer your question. Also, what do you mean by "The **continuous actions** starts of nice"? Please, also put your **specific** question in the title to immediately clarify what it is.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-05T15:26:46.250Z
Comments.Text: Hello @nbro, I have added information directly into to the question. Sorry about that!
Comments.UserDisplayName: hridayns
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-05T15:31:14.007Z
Comments.Text: Thanks, but please tells us also what your main question is. So, it's the loss that becomes `nan`. So, that's what you mean by "continuous actions", i.e. the loss?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-05T15:36:26.070Z
Comments.Text: @nbro No, my continuous actions also become nan! I do not know the cause and effect though. Which one became nan first and corrupted the other, I do not know and I am trying to figure that out as well.
Comments.UserDisplayName: hridayns
Tags: <reinforcement-learning><actor-critic-methods><proximal-policy-optimization><multi-agent-systems><continuous-action-spaces>
Title: RLlib's Multi-agent PPO continuous actions turn into nan
UserDisplayName: hridayns
ThreadText: # RLlib's Multi-agent PPO continuous actions turn into nan
  
  (Asked by: username_0 on 2021-07-05)
  
  After some amount of training on a custom Multi-agent sparse-reward environment using RLlib's (1.4.0) PPO network, I found that my continuous actions turn into nan (explodes?) which is probably caused by a bad gradient update which in turn depends on the loss/objective function.
  
  As I understand it, PPO's loss function relies on three terms:
  
  1. The **clipped surrogate objective** which depends on outputs of old policy and new policy, the advantage, and the "clip" parameter(=0.3)
  2. The **Value Function Loss**
  3. The **Entropy Loss** [mainly there to encourage exploration]
  
  **Total Loss** = **Surrogate objective (clipped)** - vf\_loss\_coeff \* **VF Loss** + entropy\_coeff \* **entropy**.
  
  [![enter image description here](https://i.sstatic.net/RxmqJ.png)](https://i.sstatic.net/RxmqJ.png)
  
  The surrogate loss
  ( Reference: <https://arxiv.org/abs/1707.06347> )
  
  I have a bunch of questions:
  
  1. Is the **ratio rt(theta)** used that of the actual actions taken from new policy vs. old policy or is it the **probability distributions** of those actions? (since actions are continuous?)
  2. Follow up question to 1: Assuming it is probability, **can the probability ever be 0?** Because if it is ever 0, then log probability would result in log(0) = inf/undefined - which would prove that is the root cause?
  3. If 1 and 2 are safely debunked, then do I
  
  **(A)** lower my learning rate?
  
  **(B)** Reduce the number of layers in my network?
  
  **(C)** Use gradient clipping or action or reward clipping of some sort?
  
  To anyone who would be kind enough to share any insights into the matter, you have my gratitude.
  
  For more information, see relevant part of **progress table below**: where the **total loss becomes inf**. The only change I found is that the policy loss was all negative until row **#445**.
  
  |  | Total loss | policy loss | VF loss |
  | --- | --- | --- | --- |
  | 430 | 6.068537 | -0.053691725999999995 | 6.102932 |
  | 431 | 5.9919114 | -0.046943977000000005 | 6.0161843 |
  | 432 | 8.134636 | -0.05247503 | 8.164852 |
  | 433 | 4.222730599999999 | -0.048518334 | 4.2523246 |
  | 434 | 6.563492 | -0.05237444 | 6.594456 |
  | 435 | 8.171028999999999 | -0.048245672 | 8.198222999999999 |
  | 436 | 8.948264 | -0.048484523 | 8.976327000000001 |
  | 437 | 7.556602000000001 | -0.054372005 | 7.5880575 |
  | 438 | 6.124418 | -0.05249534 | 6.155608999999999 |
  | 439 | 4.267647 | -0.052565258 | 4.2978816 |
  | 440 | 4.912957700000001 | -0.054498855 | 4.9448576 |
  | 441 | 16.630292999999998 | -0.043477765999999994 | 16.656229 |
  | 442 | 6.3149705 | -0.057527818 | 6.349851999999999 |
  | 443 | 4.2269225 | -0.05446908599999999 | 4.260793700000001 |
  | 444 | 9.503102 | -0.052135203 | 9.53277 |
  | 445 | **inf** | **0.2436709** | **4.410831** |
  | 446 | nan | -0.00029848056 | 22.596403 |
  | 447 | nan | 0.00013323531 | 0.00043436907999999994 |
  | 448 | nan | 1.5656527000000002e-05 | 0.0002645221 |
  | 449 | nan | 1.3344318000000001e-05 | 0.0003139485 |
  | 450 | nan | 6.941916999999999e-05 | 0.00025863337 |
  | 451 | nan | 0.00015686743 | 0.00013607396 |
  | 452 | nan | -5.0206604e-06 | 0.00027541115000000003 |
  | 453 | nan | -4.5543664e-05 | 0.0004247162 |
  | 454 | nan | 8.841756999999999e-05 | 0.00020278389999999998 |
  | 455 | nan | -8.465959e-05 | 9.261127e-05 |
  | 456 | nan | 3.8680790000000003e-05 | 0.00032097592999999995 |
  | 457 | nan | 2.7373152999999996e-06 | 0.0005146417 |
  | 458 | nan | -6.271608e-06 | 0.0013273798000000001 |
  | 459 | nan | -0.00013192794 | 0.00030621013 |
  | 460 | nan | 0.00038987884 | 0.00038019830000000004 |
  | 461 | nan | -3.2747877999999998e-06 | 0.00031471922 |
  | 462 | nan | -6.9349815e-05 | 0.00038836736000000006 |
  | 463 | nan | -4.666238e-05 | 0.0002851575 |
  | 464 | nan | -3.7067155e-05 | 0.00020161088 |
  | 465 | nan | 3.0623291e-06 | 0.00019258813999999998 |
  | 466 | nan | -8.599938e-06 | 0.00036465342000000005 |
  | 467 | nan | -1.1529375e-05 | 0.00016500981 |
  | 468 | nan | -3.0851965e-07 | 0.00022042097 |
  | 469 | nan | -0.0001133984 | 0.00030230957999999997 |
  | 470 | nan | -1.0735256e-05 | 0.00034000343000000003 |
  
  **Optional**
  
  For even further context, check my related [question](https://stackoverflow.com/questions/68249391/rllib-ppo-continuous-actions-seem-to-become-nan-after-total-loss-inf)
  
  **Tags:** reinforcement-learning, actor-critic-methods, proximal-policy-optimization, multi-agent-systems, continuous-action-spaces
  
  > **Comments:**
  > **username_1**: Hello. Please, do not just link us to another external post/article to provide/describe the context. Make sure you include in this post all the information needed to answer your question. Also, what do you mean by "The **continuous actions** starts of nice"? Please, also put your **specific** question in the title to immediately clarify what it is.  
  > **username_0**: Hello @username_1, I have added information directly into to the question. Sorry about that!  
  > **username_1**: Thanks, but please tells us also what your main question is. So, it's the loss that becomes `nan`. So, that's what you mean by "continuous actions", i.e. the loss?  
  > **username_0**: @username_1 No, my continuous actions also become nan! I do not know the cause and effect though. Which one became nan first and corrupted the other, I do not know and I am trying to figure that out as well.  
  
  ---
  
  ## 0 answers
  
  