Answers.Body: <p>I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T12:36:08.580Z
Answers.LastActivityDate: 2019-07-25T12:36:08.580Z
Answers.UserDisplayName: Chintan Trivedi
Body: <p>I have implemented the total loss of my PPO objective as follows:-</p>
  
  <pre><code>total_loss = critic_discount * critic_loss + actor_loss -  entropy_beta * K.mean(-(newpolicy_probs * K.log(newpolicy_probs)))
  </code></pre>
  
  <p>After training for a few epochs, the entropy term becomes "nan" for some reason. I used <code>tf.Print()</code> to see the new policy probabilities when the entropy becomes undefined, it is as follows-</p>
  
  <blockquote>
    <p>new policy probs: [[6.1029973e-06 1.93471514e-08
    0.000299338106...]...]</p>
  </blockquote>
  
  <p>I am not clear as to why taking log of these small probabilities is coming out as <code>nan</code>. Any idea how to prevent this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-24T20:18:34.203Z
Comments.Text: Log of zero is mathematically undefined. On a computer, the log is likely approximated (it depends on the implementation). Furthermore, the multiplication of small numbers creates even smaller numbers (e.g. the product of $0.1*0.1 = 0.01$), which leads quite quickly to the "vanishing problem". Are you using floats (single-precision) or doubles? If you're using floats, maybe try using doubles? Or maybe try to manually handle the case where too small probabilities that are passed to the log. If this answers your question, let me know, and I will convert this comment to an answer.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-24T20:21:20.813Z
Comments.Text: However, note that this question is off-topic here. You should ask it on Stack Overflow, given that this is a problem related to the implementation. Here we focus more on conceptual and philosophical questions.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-25T14:14:33.150Z
Comments.Text: I addressed my implemetation issue with the solution in my answer below. However, I still have conceptual doubts so I opened a separate question here: https://ai.stackexchange.com/questions/13577/understanding-log-probabilities-of-actions-in-ppo-objective , thanks for the advice!
Comments.UserDisplayName: Chintan Trivedi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-25T14:17:43.427Z
Comments.Text: I did handle small values manually by adding 1e-10 so if you want to post it as an answer I'll accept it.
Comments.UserDisplayName: Chintan Trivedi
Tags: <reinforcement-learning><python><keras><proximal-policy-optimization>
Title: Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs
UserDisplayName: Chintan Trivedi
ThreadText: # Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs
  
  (Asked by: username_0 on 2019-07-24)
  
  I have implemented the total loss of my PPO objective as follows:-
  
  ```
  total_loss = critic_discount * critic_loss + actor_loss -  entropy_beta * K.mean(-(newpolicy_probs * K.log(newpolicy_probs)))
  
  ```
  
  After training for a few epochs, the entropy term becomes "nan" for some reason. I used `tf.Print()` to see the new policy probabilities when the entropy becomes undefined, it is as follows-
  
  > new policy probs: [[6.1029973e-06 1.93471514e-08
  > 0.000299338106...]...]
  
  I am not clear as to why taking log of these small probabilities is coming out as `nan`. Any idea how to prevent this?
  
  **Tags:** reinforcement-learning, python, keras, proximal-policy-optimization
  
  > **Comments:**
  > **username_1**: Log of zero is mathematically undefined. On a computer, the log is likely approximated (it depends on the implementation). Furthermore, the multiplication of small numbers creates even smaller numbers (e.g. the product of $0.1*0.1 = 0.01$), which leads quite quickly to the "vanishing problem". Are you using floats (single-precision) or doubles? If you're using floats, maybe try using doubles? Or maybe try to manually handle the case where too small probabilities that are passed to the log. If this answers your question, let me know, and I will convert this comment to an answer.  
  > **username_1**: However, note that this question is off-topic here. You should ask it on Stack Overflow, given that this is a problem related to the implementation. Here we focus more on conceptual and philosophical questions.  
  > **username_0**: I addressed my implemetation issue with the solution in my answer below. However, I still have conceptual doubts so I opened a separate question here: https://ai.stackexchange.com/questions/13577/understanding-log-probabilities-of-actions-in-ppo-objective , thanks for the advice!  
  > **username_0**: I did handle small values manually by adding 1e-10 so if you want to post it as an answer I'll accept it.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-07-25. Score: 3
  
  I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.
  
  
  ---
  
  