How to deal with negative rewards in policy gradient with crossentropy loss In policy gradient reinforcement learning we can use a loss function of the form -log(P)*reward, where P is the probability of the selected action given the policy.  For discrete actions, this turns into categorical_crossentropy:
Loss = categorical_crossentropy(chosen_action, policy_output)*reward

Some people seem to think this is just fine with negative rewards, but others not. In application, the issue I'm seeing is that hugely negative losses can be achieved. If the reward is, say, -1.0, then the loss is unboundedly negative. If the policy can be pushed to nearly zero for the chosen action, the loss can go nearly to minus infinity. For a reward of +1.0, the loss can only go down to zero.  I think this asymmetry is problematic.
How should this be managed?


The loss functions are indeed asymmetric. But the derivatives of the loss functions are similar in both of the cases. If you have a look at the contour plot of the derivative of binary-cross entropy, you will observe that it is symmetric along the x=y line (they have the same magnitude but of opposite sign). As the gradient is similar, their effect on the learning algorithm is also similar.
