Body: <p>My problem is that every time I am trying to train my PPO agent I get NaN values after a while. The diagnostic that I get is the following:</p>
  
  <pre><code>ep=   3| t=  144/450000|  0.8 sec| rew=31.3005| ploss=-2.5e-02| vfloss=4.7e+01| kl=1.3e-03| ent=8.5e+00| clipfrac=0.0e+00
  ep=   6| t=  288/450000|  1.1 sec| rew=31.2144| ploss=-2.2e-02| vfloss=4.1e+01| kl=1.3e-03| ent=8.5e+00| clipfrac=0.0e+00
  ep=   9| t=  432/450000|  1.4 sec| rew=28.2668| ploss=-2.9e-02| vfloss=3.5e+01| kl=1.6e-03| ent=8.5e+00| clipfrac=0.0e+00
  ep=  12| t=  576/450000|  1.7 sec| rew=28.2910| ploss=-2.7e-02| vfloss=3.6e+01| kl=1.7e-03| ent=8.5e+00| clipfrac=0.0e+00
  ep=  15| t=  720/450000|  2.0 sec| rew=27.4817| ploss=-2.3e-02| vfloss=3.0e+01| kl=1.8e-03| ent=8.5e+00| clipfrac=0.0e+00
  ep=  18| t=  864/450000|  2.3 sec| rew=29.8415| ploss=-4.5e-02| vfloss=3.4e+01| kl=4.0e-03| ent=8.5e+00| clipfrac=2.8e-02
  ep=  21| t= 1008/450000|  2.6 sec| rew=29.1447| ploss=-2.7e-02| vfloss=2.7e+01| kl=2.0e-03| ent=8.5e+00| clipfrac=6.9e-03
  ep=  24| t= 1152/450000|  3.0 sec| rew=30.2001| ploss=-3.5e-02| vfloss=2.8e+01| kl=1.7e-03| ent=8.5e+00| clipfrac=6.9e-03
  ep=  27| t= 1296/450000|  3.3 sec| rew=31.4069| ploss=-2.9e-02| vfloss=3.7e+01| kl=3.0e-03| ent=8.5e+00| clipfrac=2.1e-02
  ep=  30| t= 1440/450000|  3.6 sec| rew=27.7963| ploss=-4.6e-02| vfloss=2.3e+01| kl=7.3e-03| ent=8.5e+00| clipfrac=1.7e-01
  ep=  33| t= 1584/450000|  3.9 sec| rew=30.8561| ploss=-5.9e-02| vfloss=2.5e+01| kl=9.6e-03| ent=8.5e+00| clipfrac=2.6e-01
  ep=  36| t= 1728/450000|  4.2 sec| rew=27.3002| ploss=-6.9e-02| vfloss=2.2e+01| kl=1.3e-02| ent=8.5e+00| clipfrac=3.1e-01
  ep=  39| t= 1872/450000|  4.5 sec| rew=28.0270| ploss=-5.6e-02| vfloss=2.1e+01| kl=8.9e-03| ent=8.5e+00| clipfrac=2.0e-01
  ep=  42| t= 2016/450000|  4.9 sec| rew=28.0624| ploss=-5.8e-02| vfloss=2.0e+01| kl=7.5e-03| ent=8.5e+00| clipfrac=2.4e-01
  ep=  45| t= 2160/450000|  5.2 sec| rew=28.6224| ploss=-8.4e-02| vfloss=2.3e+01| kl=7.2e-03| ent=8.5e+00| clipfrac=2.0e-01
  ep=  48| t= 2304/450000|  5.5 sec| rew=32.3889| ploss=-4.3e-02| vfloss=2.6e+01| kl=7.1e-03| ent=8.5e+00| clipfrac=2.9e-01
  ep=  51| t= 2448/450000|  5.8 sec| rew=31.4241| ploss=-1.0e-01| vfloss=2.7e+01| kl=7.0e-03| ent=8.5e+00| clipfrac=2.1e-01
  ep=  54| t= 2592/450000|  6.1 sec| rew=33.4760| ploss=-5.1e-02| vfloss=2.5e+01| kl=7.3e-03| ent=8.5e+00| clipfrac=2.4e-01
  ep=  57| t= 2736/450000|  6.4 sec| rew=31.0780| ploss=-8.8e-02| vfloss=2.3e+01| kl=6.9e-03| ent=8.5e+00| clipfrac=3.0e-01
  ep=  60| t= 2880/450000|  6.7 sec| rew=34.1286| ploss=-6.9e-02| vfloss=2.7e+01| kl=7.6e-03| ent=8.5e+00| clipfrac=3.1e-01
  ep=  63| t= 3024/450000|  7.1 sec| rew=31.0017| ploss=-6.1e-02| vfloss=2.5e+01| kl=1.4e-02| ent=8.5e+00| clipfrac=3.7e-01
  ep=  66| t= 3168/450000|  7.4 sec| rew=32.3697| ploss=-1.1e-01| vfloss=2.2e+01| kl=1.2e-02| ent=8.5e+00| clipfrac=4.6e-01
  ep=  69| t= 3312/450000|  7.7 sec| rew=31.4455| ploss=-7.4e-02| vfloss=2.4e+01| kl=8.4e-03| ent=8.5e+00| clipfrac=3.7e-01
  ep=  72| t= 3456/450000|  8.0 sec| rew=32.1896| ploss=-9.2e-02| vfloss=2.0e+01| kl=1.3e-02| ent=8.4e+00| clipfrac=4.1e-01
  ep=  75| t= 3600/450000|  8.3 sec| rew=31.3721| ploss=-9.4e-02| vfloss=2.4e+01| kl=1.4e-02| ent=8.4e+00| clipfrac=4.2e-01
  ep=  78| t= 3744/450000|  8.6 sec| rew=35.5718| ploss=-1.0e-01| vfloss=3.0e+01| kl=1.0e-02| ent=8.4e+00| clipfrac=4.6e-01
  ep=  81| t= 3888/450000|  9.0 sec| rew=32.2289| ploss=-1.3e-01| vfloss=2.9e+01| kl=1.6e-02| ent=8.4e+00| clipfrac=5.5e-01
  ep=  84| t= 4032/450000|  9.3 sec| rew=31.7656| ploss=-1.0e-01| vfloss=2.3e+01| kl=1.3e-02| ent=8.4e+00| clipfrac=4.4e-01
  ep=  87| t= 4176/450000|  9.6 sec| rew=35.4555| ploss=-8.8e-02| vfloss=3.3e+01| kl=5.8e-03| ent=8.4e+00| clipfrac=3.1e-01
  ep=  90| t= 4320/450000|  9.9 sec| rew=33.2766| ploss=-1.2e-01| vfloss=2.4e+01| kl=1.5e-02| ent=8.4e+00| clipfrac=5.7e-01
  ep=  93| t= 4464/450000| 10.2 sec| rew=32.5218| ploss=-1.1e-01| vfloss=2.4e+01| kl=1.8e-02| ent=8.4e+00| clipfrac=5.6e-01
  ep=  96| t= 4608/450000| 10.5 sec| rew=34.7137| ploss=-9.7e-02| vfloss=2.5e+01| kl=1.5e-02| ent=8.4e+00| clipfrac=4.3e-01
  ep=  99| t= 4752/450000| 10.9 sec| rew=35.3797| ploss=-8.2e-02| vfloss=2.5e+01| kl=1.6e-02| ent=8.4e+00| clipfrac=4.9e-01
  ep= 102| t= 4896/450000| 11.2 sec| rew=nan| ploss=nan| vfloss=nan| kl=nan| ent=nan| clipfrac=0.0e+00
  C:\Users\ppo.py:154: RuntimeWarning: invalid value encountered in greater
    adv_nrm = (adv_nrm - adv_nrm.mean()) / max(1.e-8,adv_nrm.std()) # standardized advantage function estimate
  ep= 105| t= 5040/450000| 11.5 sec| rew=nan| ploss=nan| vfloss=nan| kl=nan| ent=nan| clipfrac=0.0e+00
  </code></pre>
  
  <p>Any ideas why this is arised? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-11T10:58:49.503Z
Comments.Text: $\log (0) \cdot 0$ probably, add little constant to argument in $\log$
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-05T01:23:30.793Z
Comments.Text: Sorry but did you figure this out?
Comments.UserDisplayName: hridayns
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-09T17:20:44.730Z
Comments.Text: @hridayns Sorry for the late response. I remember that I changed the implementation of the PPO algorithm that I was using to the one and it worked out fine.
Comments.UserDisplayName: Kosmylo
Tags: <reinforcement-learning><deep-rl><proximal-policy-optimization>
Title: NaNs after a while in training of PPO
UserDisplayName: Kosmylo
ThreadText: # NaNs after a while in training of PPO
  
  (Asked by: username_0 on 2020-04-11)
  
  My problem is that every time I am trying to train my PPO agent I get NaN values after a while. The diagnostic that I get is the following:
  
  ```
  ep=   3| t=  144/450000|  0.8 sec| rew=31.3005| ploss=-2.5e-02| vfloss=4.7e+01| kl=1.3e-03| ent=8.5e+00| clipfrac=0.0e+00
  ep=   6| t=  288/450000|  1.1 sec| rew=31.2144| ploss=-2.2e-02| vfloss=4.1e+01| kl=1.3e-03| ent=8.5e+00| clipfrac=0.0e+00
  ep=   9| t=  432/450000|  1.4 sec| rew=28.2668| ploss=-2.9e-02| vfloss=3.5e+01| kl=1.6e-03| ent=8.5e+00| clipfrac=0.0e+00
  ep=  12| t=  576/450000|  1.7 sec| rew=28.2910| ploss=-2.7e-02| vfloss=3.6e+01| kl=1.7e-03| ent=8.5e+00| clipfrac=0.0e+00
  ep=  15| t=  720/450000|  2.0 sec| rew=27.4817| ploss=-2.3e-02| vfloss=3.0e+01| kl=1.8e-03| ent=8.5e+00| clipfrac=0.0e+00
  ep=  18| t=  864/450000|  2.3 sec| rew=29.8415| ploss=-4.5e-02| vfloss=3.4e+01| kl=4.0e-03| ent=8.5e+00| clipfrac=2.8e-02
  ep=  21| t= 1008/450000|  2.6 sec| rew=29.1447| ploss=-2.7e-02| vfloss=2.7e+01| kl=2.0e-03| ent=8.5e+00| clipfrac=6.9e-03
  ep=  24| t= 1152/450000|  3.0 sec| rew=30.2001| ploss=-3.5e-02| vfloss=2.8e+01| kl=1.7e-03| ent=8.5e+00| clipfrac=6.9e-03
  ep=  27| t= 1296/450000|  3.3 sec| rew=31.4069| ploss=-2.9e-02| vfloss=3.7e+01| kl=3.0e-03| ent=8.5e+00| clipfrac=2.1e-02
  ep=  30| t= 1440/450000|  3.6 sec| rew=27.7963| ploss=-4.6e-02| vfloss=2.3e+01| kl=7.3e-03| ent=8.5e+00| clipfrac=1.7e-01
  ep=  33| t= 1584/450000|  3.9 sec| rew=30.8561| ploss=-5.9e-02| vfloss=2.5e+01| kl=9.6e-03| ent=8.5e+00| clipfrac=2.6e-01
  ep=  36| t= 1728/450000|  4.2 sec| rew=27.3002| ploss=-6.9e-02| vfloss=2.2e+01| kl=1.3e-02| ent=8.5e+00| clipfrac=3.1e-01
  ep=  39| t= 1872/450000|  4.5 sec| rew=28.0270| ploss=-5.6e-02| vfloss=2.1e+01| kl=8.9e-03| ent=8.5e+00| clipfrac=2.0e-01
  ep=  42| t= 2016/450000|  4.9 sec| rew=28.0624| ploss=-5.8e-02| vfloss=2.0e+01| kl=7.5e-03| ent=8.5e+00| clipfrac=2.4e-01
  ep=  45| t= 2160/450000|  5.2 sec| rew=28.6224| ploss=-8.4e-02| vfloss=2.3e+01| kl=7.2e-03| ent=8.5e+00| clipfrac=2.0e-01
  ep=  48| t= 2304/450000|  5.5 sec| rew=32.3889| ploss=-4.3e-02| vfloss=2.6e+01| kl=7.1e-03| ent=8.5e+00| clipfrac=2.9e-01
  ep=  51| t= 2448/450000|  5.8 sec| rew=31.4241| ploss=-1.0e-01| vfloss=2.7e+01| kl=7.0e-03| ent=8.5e+00| clipfrac=2.1e-01
  ep=  54| t= 2592/450000|  6.1 sec| rew=33.4760| ploss=-5.1e-02| vfloss=2.5e+01| kl=7.3e-03| ent=8.5e+00| clipfrac=2.4e-01
  ep=  57| t= 2736/450000|  6.4 sec| rew=31.0780| ploss=-8.8e-02| vfloss=2.3e+01| kl=6.9e-03| ent=8.5e+00| clipfrac=3.0e-01
  ep=  60| t= 2880/450000|  6.7 sec| rew=34.1286| ploss=-6.9e-02| vfloss=2.7e+01| kl=7.6e-03| ent=8.5e+00| clipfrac=3.1e-01
  ep=  63| t= 3024/450000|  7.1 sec| rew=31.0017| ploss=-6.1e-02| vfloss=2.5e+01| kl=1.4e-02| ent=8.5e+00| clipfrac=3.7e-01
  ep=  66| t= 3168/450000|  7.4 sec| rew=32.3697| ploss=-1.1e-01| vfloss=2.2e+01| kl=1.2e-02| ent=8.5e+00| clipfrac=4.6e-01
  ep=  69| t= 3312/450000|  7.7 sec| rew=31.4455| ploss=-7.4e-02| vfloss=2.4e+01| kl=8.4e-03| ent=8.5e+00| clipfrac=3.7e-01
  ep=  72| t= 3456/450000|  8.0 sec| rew=32.1896| ploss=-9.2e-02| vfloss=2.0e+01| kl=1.3e-02| ent=8.4e+00| clipfrac=4.1e-01
  ep=  75| t= 3600/450000|  8.3 sec| rew=31.3721| ploss=-9.4e-02| vfloss=2.4e+01| kl=1.4e-02| ent=8.4e+00| clipfrac=4.2e-01
  ep=  78| t= 3744/450000|  8.6 sec| rew=35.5718| ploss=-1.0e-01| vfloss=3.0e+01| kl=1.0e-02| ent=8.4e+00| clipfrac=4.6e-01
  ep=  81| t= 3888/450000|  9.0 sec| rew=32.2289| ploss=-1.3e-01| vfloss=2.9e+01| kl=1.6e-02| ent=8.4e+00| clipfrac=5.5e-01
  ep=  84| t= 4032/450000|  9.3 sec| rew=31.7656| ploss=-1.0e-01| vfloss=2.3e+01| kl=1.3e-02| ent=8.4e+00| clipfrac=4.4e-01
  ep=  87| t= 4176/450000|  9.6 sec| rew=35.4555| ploss=-8.8e-02| vfloss=3.3e+01| kl=5.8e-03| ent=8.4e+00| clipfrac=3.1e-01
  ep=  90| t= 4320/450000|  9.9 sec| rew=33.2766| ploss=-1.2e-01| vfloss=2.4e+01| kl=1.5e-02| ent=8.4e+00| clipfrac=5.7e-01
  ep=  93| t= 4464/450000| 10.2 sec| rew=32.5218| ploss=-1.1e-01| vfloss=2.4e+01| kl=1.8e-02| ent=8.4e+00| clipfrac=5.6e-01
  ep=  96| t= 4608/450000| 10.5 sec| rew=34.7137| ploss=-9.7e-02| vfloss=2.5e+01| kl=1.5e-02| ent=8.4e+00| clipfrac=4.3e-01
  ep=  99| t= 4752/450000| 10.9 sec| rew=35.3797| ploss=-8.2e-02| vfloss=2.5e+01| kl=1.6e-02| ent=8.4e+00| clipfrac=4.9e-01
  ep= 102| t= 4896/450000| 11.2 sec| rew=nan| ploss=nan| vfloss=nan| kl=nan| ent=nan| clipfrac=0.0e+00
  C:\Users\ppo.py:154: RuntimeWarning: invalid value encountered in greater
    adv_nrm = (adv_nrm - adv_nrm.mean()) / max(1.e-8,adv_nrm.std()) # standardized advantage function estimate
  ep= 105| t= 5040/450000| 11.5 sec| rew=nan| ploss=nan| vfloss=nan| kl=nan| ent=nan| clipfrac=0.0e+00
  
  ```
  
  Any ideas why this is arised?
  
  **Tags:** reinforcement-learning, deep-rl, proximal-policy-optimization
  
  > **Comments:**
  > **username_1**: $\log (0) \cdot 0$ probably, add little constant to argument in $\log$  
  > **username_2**: Sorry but did you figure this out?  
  > **username_0**: @username_2 Sorry for the late response. I remember that I changed the implementation of the PPO algorithm that I was using to the one and it worked out fine.  
  
  ---
  
  ## 0 answers
  
  