Answers.Body: <p>You don't need to manage negative rewards separately, if you implemented the algorithm correctly it will work regardless if the rewards are negative or not. You seem to be using rewards for the loss but you should be using the return which is the sum of the rewards for some state action pair from that point until the end of trajectory.</p>
  
  <p>You also seem to be missing <span class="math-container">$-$</span> sign from the loss. The objective function for the vanilla policy gradient algorithm (REINFORCE) which we want to maximize is
  <span class="math-container">\begin{equation}
  J = \sum_a \pi(a|s) q_{\pi}(s, a)
  \end{equation}</span>
  It can be shown that the gradient sample for this policy gradient method is
  <span class="math-container">\begin{equation}
  \nabla J = G_t \nabla \log (\pi(A_t|S_t))
  \end{equation}</span>
  so in TensorFlow you should define your loss as
  <span class="math-container">\begin{equation}
  J = - G_t \pi(A_t|S_t)
  \end{equation}</span>
  We need the <span class="math-container">$-$</span> because in TensorFlow you use minimizers, but we want to maximize this function so minimizing this loss is same as maximizing the objective function. In conclusion, the code similar to what you wrote, should be<br>
  <code>-return * cross_entropy(action_pdf, selected_action_one_hot)</code></p>
  
  <p><strong>EDIT</strong></p>
  
  <p>As pointed out in the comment we don't actually need <span class="math-container">$-$</span> because it is already included in <code>cross_entropy</code> function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-18T18:39:13.967Z
Answers.Text: Thanks.  I'm not sure about the sign as you have it.  cross_entropy loss in keras has an implied negative sign relative to the normal definition of crossentropy.  My models train without the minus sign.  See also: https://stackoverflow.com/a/56893454/2364295
Answers.UserDisplayName: Mastiff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-18T18:42:18.300Z
Answers.Text: But regardless of the above.  My issue is with negative rewards (or returns).  I implement exploration by deviating from the model PDF output in some cases.  If the model outputs nearly 100% confident actions and the exploration logic deviates, a negative return can lead to a hugely negative reward.  Perhaps my exploration approach is invalid.
Answers.UserDisplayName: Mastiff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-18T19:11:48.953Z
Answers.Text: You're right about the $-$, I edited the answer, as for the returns are you saying that the model drops performance when you choose an action that's not the one given by the model ? How do you choose it, do you sample it from the output distribution ?
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-18T19:16:14.197Z
Answers.Text: Also, make sure you ciip the gradients, because the larger return will cause large parameter updates and the policy may break with too large updates because it will deviate from the current one too much.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-18T19:50:40.940Z
Answers.Text: When training I allow the model to explore by doing a proportional draw from the model provided softmax PDF.  The detail is that I also broaden the PDF a bit from the model version so it doesn't collapse and stop exploring (by adding a small value and renormalizing).  If the action I choose happens to have zero probability from the model, and negative return, the loss will be hugely negative.  I'm experimenting now with putting the PDF softening into the model itself, controlled by an input, so I can have the selected action always be consistent with the model.
Answers.UserDisplayName: Mastiff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-18T17:22:31.803Z
Answers.LastActivityDate: 2020-05-18T19:11:50.887Z
Answers.LastEditDate: 2020-05-18T19:11:50.887Z
Answers.UserDisplayName: Brale
Body: <p><a href="https://ai.stackexchange.com/q/2405/2444">This old question</a> has no definitive answer yet, that's why I am asking it here again. I also asked this same question <a href="https://stackoverflow.com/q/61829580/3924118">here</a>.</p>
  <p>If I'm doing policy gradient in Keras, using a loss of the form:</p>
  <pre><code>rewards*cross_entropy(action_pdf, selected_action_one_hot)
  </code></pre>
  <p>How do I manage negative rewards?</p>
  <p>I've had success with this form in cases where the reward is always positive, but it does not train with negative rewards. The failure mode is for it to drive itself to very confident predictions all the time, which results in very large negative losses due to induced deviation for exploration. I can get it to train by clipping rewards at zero, but this throws a lot of valuable information on the table (only carrots, no sticks).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-01T22:52:03.640Z
Comments.Text: If you have a question that has already been asked but no good answer has been given, one way to drive attention to that question is to [start a bounty](https://stackoverflow.com/help/bounty).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><keras><policy-gradients><rewards><cross-entropy>
Title: How do you manage negative rewards in policy gradients?
UserDisplayName: Mastiff
ThreadText: # How do you manage negative rewards in policy gradients?
  
  (Asked by: username_0 on 2020-05-18)
  
  This old question has no definitive answer yet, that's why I am asking it here again. I also asked this same question here.
  If I'm doing policy gradient in Keras, using a loss of the form:
  rewards*cross_entropy(action_pdf, selected_action_one_hot)
  
  How do I manage negative rewards?
  I've had success with this form in cases where the reward is always positive, but it does not train with negative rewards. The failure mode is for it to drive itself to very confident predictions all the time, which results in very large negative losses due to induced deviation for exploration. I can get it to train by clipping rewards at zero, but this throws a lot of valuable information on the table (only carrots, no sticks).
  
  **Tags:** reinforcement-learning, keras, policy-gradients, rewards, cross-entropy
  > **Comments:**
  > **username_1**: If you have a question that has already been asked but no good answer has been given, one way to drive attention to that question is to [start a bounty](https://stackoverflow.com/help/bounty).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-05-18. Score: 1
  
  You don't need to manage negative rewards separately, if you implemented the algorithm correctly it will work regardless if the rewards are negative or not. You seem to be using rewards for the loss but you should be using the return which is the sum of the rewards for some state action pair from that point until the end of trajectory.
  
  You also seem to be missing $-$ sign from the loss. The objective function for the vanilla policy gradient algorithm (REINFORCE) which we want to maximize is
  \begin{equation}
  J = \sum\_a \pi(a|s) q\_{\pi}(s, a)
  \end{equation}
  It can be shown that the gradient sample for this policy gradient method is
  \begin{equation}
  \nabla J = G\_t \nabla \log (\pi(A\_t|S\_t))
  \end{equation}
  so in TensorFlow you should define your loss as
  \begin{equation}
  J = - G\_t \pi(A\_t|S\_t)
  \end{equation}
  We need the $-$ because in TensorFlow you use minimizers, but we want to maximize this function so minimizing this loss is same as maximizing the objective function. In conclusion, the code similar to what you wrote, should be  
  `-return * cross_entropy(action_pdf, selected_action_one_hot)`
  
  **EDIT**
  
  As pointed out in the comment we don't actually need $-$ because it is already included in `cross_entropy` function.
  
  > **Comments:**
  > **username_0**: Thanks.  I'm not sure about the sign as you have it.  cross_entropy loss in keras has an implied negative sign relative to the normal definition of crossentropy.  My models train without the minus sign.  See also: https://stackoverflow.com/a/56893454/2364295  
  > **username_0**: But regardless of the above.  My issue is with negative rewards (or returns).  I implement exploration by deviating from the model PDF output in some cases.  If the model outputs nearly 100% confident actions and the exploration logic deviates, a negative return can lead to a hugely negative reward.  Perhaps my exploration approach is invalid.  
  > **username_2**: You're right about the $-$, I edited the answer, as for the returns are you saying that the model drops performance when you choose an action that's not the one given by the model ? How do you choose it, do you sample it from the output distribution ?  
  > **username_2**: Also, make sure you ciip the gradients, because the larger return will cause large parameter updates and the policy may break with too large updates because it will deviate from the current one too much.  
  > **username_0**: When training I allow the model to explore by doing a proportional draw from the model provided softmax PDF.  The detail is that I also broaden the PDF a bit from the model version so it doesn't collapse and stop exploring (by adding a small value and renormalizing).  If the action I choose happens to have zero probability from the model, and negative return, the loss will be hugely negative.  I'm experimenting now with putting the PDF softening into the model itself, controlled by an input, so I can have the selected action always be consistent with the model.  
  
  ---
  
  