Answers.Body: <p>It depends on your loss function, but you probably need to tweak it. </p>
  
  <p>If you are using an update rule like <code>loss = -log(probabilities) * reward</code>, then your loss is high when you unexpectedly got a large reward—the policy will update to make that action more likely to realize that gain.</p>
  
  <p>Conversely, if you get a negative reward with high probability, this will result in negative loss—however, in minimizing this loss, the optimizer will attempt to make this loss "even more negative" by making the log probability more negative (i.e. by making the probability of that action less likely)—so it kind of does what we want. </p>
  
  <p>However, now improbable large negative losses are punished more than the more than likely ones, when we probably want the opposite. Hence, <code>loss = -log(1-probabilities) * reward</code> might be more appropriate when the reward is negative.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-13T04:47:34.273Z
Answers.Text: I think you meant `loss = -log(probabilities) * reward` (minus before the log)
Answers.UserDisplayName: Zsolt Safrany
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-13T21:00:57.120Z
Answers.Text: @user12889 I'm thinking it is symmetric in the sense that a -1 reward has precisely the opposite gradient of a +1 reward. I think the asymmetry you're describing is related to using negative log probabilities as opposed to using something like 1-p. But this asymmetry of the loss function will impact both positive and negative rewards the same.
Answers.UserDisplayName: Tahlor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-14T22:27:36.523Z
Answers.Text: @Tahlor Yes, but over time, as the model learns, you would expect probabilities for +1 rewards to move closer to 1 and probabilities for -1 rewards to move closer to 0, which then leads to generally highly non-symmetric gradients for +1 and -1 rewards.
Answers.UserDisplayName: user12889
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-15T01:17:52.830Z
Answers.Text: Oh right, so couldn't you just invert and shift your loss function for negative rewards?
  So `-log(1-probabilities) * reward` for negative rewards and the regular one for positive rewards to preserve that symmetry?
Answers.UserDisplayName: Tahlor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-05T00:33:36.447Z
Answers.Text: While negative rewards being valid, wouldn't they really screw up your gradient - in the sense that the gradient for positive and negative rewards would be highly asymmetric? Positive rewards will cause a diminishing gradient the closer the action probability goes to 1, whereas negative rewards will cause a strongly increasing gradient the closer the action probability goes to 0.
Answers.UserDisplayName: user12889
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-04-16T21:47:46.010Z
Answers.LastActivityDate: 2019-10-16T17:23:17.077Z
Answers.LastEditDate: 2019-10-16T17:23:17.077Z
Answers.UserDisplayName: Tahlor
Answers.Body: <p>The cross-entropy loss will always be positive because the probability is in the range <span class="math-container">$[0, 1]$</span>, so <span class="math-container">$-ln(p)$</span> will always be positive.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-15T03:32:22.843Z
Answers.Text: Right, I think the issue is he's multiplying -ln(p) by a potentially negative number (his reward). This loss function expects only positive rewards.
Answers.UserDisplayName: Tahlor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-05T00:27:26.813Z
Answers.Text: @Tahlor I think you are right about the reward needing to be positive. I'm having the same problem and came to the same conclusion. However I'm having a hard time finding an authoritative source for this. Do you have one at hand?
Answers.UserDisplayName: user12889
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-10-16T15:18:47.693Z
Answers.LastActivityDate: 2020-11-01T22:39:38.373Z
Answers.LastEditDate: 2020-11-01T22:39:38.373Z
Answers.UserDisplayName: user3711746
Body: <p>I am using policy gradients in my reinforcement learning algorithm, and occasionally my environment provides a severe penalty (i.e. negative reward) when a wrong move is made. I'm using a neural network with stochastic gradient descent to learn the policy. To do this, my loss is essentially the cross-entropy loss of the action distribution multiplied by the discounted rewards, where most often the rewards are positive.</p>
  <p>But how do I handle negative rewards? Since the loss will occasionally go negative, it will think these actions are very good, and will strengthen the weights in the direction of the penalties. Is this correct, and if so, what can I do about it?</p>
  <hr />
  <p>Edit:</p>
  <p>In thinking about this a little more, SGD doesn't necessarily directly weaken weights, it only strengthens weights in the direction of the gradient and as a side-effect, weights get diminished for other states outside the gradient, correct? So I can simply set reward=0 when the reward is negative, and those states will be ignored in the gradient update. It still seems unproductive to not account for states that are really bad, and it'd be nice to include them somehow. Unless I'm misunderstanding something fundamental here.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-27T16:06:45.613Z
Comments.Text: Did you ever get to the bottom of this?  I'm still struggling with it.  The issue for me seems to be that with a negative reward, the loss can be driven to -inf in the limit.  I think the optimizer biases strongly toward driving the negative reward cases to zero, rather than pushing the positive reward cases to 1.
Comments.UserDisplayName: Mastiff
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-17T04:01:08.997Z
Comments.Text: Just clarifying, this is for discrete actions?
Comments.UserDisplayName: Jaden Travnik
Tags: <reinforcement-learning><policy-gradients><rewards><cross-entropy><stochastic-gradient-descent>
Title: How do I handle negative rewards in policy gradients with the cross-entropy loss function?
UserDisplayName: jstaker7
ThreadText: # How do I handle negative rewards in policy gradients with the cross-entropy loss function?
  
  (Asked by: username_0 on 2016-11-29)
  
  I am using policy gradients in my reinforcement learning algorithm, and occasionally my environment provides a severe penalty (i.e. negative reward) when a wrong move is made. I'm using a neural network with stochastic gradient descent to learn the policy. To do this, my loss is essentially the cross-entropy loss of the action distribution multiplied by the discounted rewards, where most often the rewards are positive.
  
  But how do I handle negative rewards? Since the loss will occasionally go negative, it will think these actions are very good, and will strengthen the weights in the direction of the penalties. Is this correct, and if so, what can I do about it?
  
  ---
  
  Edit:
  
  In thinking about this a little more, SGD doesn't necessarily directly weaken weights, it only strengthens weights in the direction of the gradient and as a side-effect, weights get diminished for other states outside the gradient, correct? So I can simply set reward=0 when the reward is negative, and those states will be ignored in the gradient update. It still seems unproductive to not account for states that are really bad, and it'd be nice to include them somehow. Unless I'm misunderstanding something fundamental here.
  
  **Tags:** reinforcement-learning, policy-gradients, rewards, cross-entropy, stochastic-gradient-descent
  > **Comments:**
  > **username_2**: Just clarifying, this is for discrete actions?  
  > **username_1**: Did you ever get to the bottom of this?  I'm still struggling with it.  The issue for me seems to be that with a negative reward, the loss can be driven to -inf in the limit.  I think the optimizer biases strongly toward driving the negative reward cases to zero, rather than pushing the positive reward cases to 1.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2018-04-16. Score: 7
  
  It depends on your loss function, but you probably need to tweak it.
  
  If you are using an update rule like `loss = -log(probabilities) * reward`, then your loss is high when you unexpectedly got a large reward—the policy will update to make that action more likely to realize that gain.
  
  Conversely, if you get a negative reward with high probability, this will result in negative loss—however, in minimizing this loss, the optimizer will attempt to make this loss "even more negative" by making the log probability more negative (i.e. by making the probability of that action less likely)—so it kind of does what we want.
  
  However, now improbable large negative losses are punished more than the more than likely ones, when we probably want the opposite. Hence, `loss = -log(1-probabilities) * reward` might be more appropriate when the reward is negative.
  
  > **Comments:**
  > **username_5**: While negative rewards being valid, wouldn't they really screw up your gradient - in the sense that the gradient for positive and negative rewards would be highly asymmetric? Positive rewards will cause a diminishing gradient the closer the action probability goes to 1, whereas negative rewards will cause a strongly increasing gradient the closer the action probability goes to 0.  
  > **username_4**: I think you meant `loss = -log(probabilities) * reward` (minus before the log)  
  > **username_3**: @user12889 I'm thinking it is symmetric in the sense that a -1 reward has precisely the opposite gradient of a +1 reward. I think the asymmetry you're describing is related to using negative log probabilities as opposed to using something like 1-p. But this asymmetry of the loss function will impact both positive and negative rewards the same.  
  > **username_5**: @Tahlor Yes, but over time, as the model learns, you would expect probabilities for +1 rewards to move closer to 1 and probabilities for -1 rewards to move closer to 0, which then leads to generally highly non-symmetric gradients for +1 and -1 rewards.  
  > **username_3**: Oh right, so couldn't you just invert and shift your loss function for negative rewards? So `-log(1-probabilities) * reward` for negative rewards and the regular one for positive rewards to preserve that symmetry?  
  
  ---
  
  ### Answer by username_6 on 2017-10-16. Score: 3
  
  The cross-entropy loss will always be positive because the probability is in the range $[0, 1]$, so $-ln(p)$ will always be positive.
  
  > **Comments:**
  > **username_3**: Right, I think the issue is he's multiplying -ln(p) by a potentially negative number (his reward). This loss function expects only positive rewards.  
  > **username_5**: @Tahlor I think you are right about the reward needing to be positive. I'm having the same problem and came to the same conclusion. However I'm having a hard time finding an authoritative source for this. Do you have one at hand?  
  
  ---
  
  