@confused, I think that the concept is that instead of comparing the loss values between your 2 cases there, you should think about minimizing each policy loss value by itself.
Let’s look at case b), in order to lower that loss, increasing the probability would lower that loss value. Now for case a), taking a step to minimize the loss would mean decreasing the probability of that action.
I wouldn’t try to view the theory as comparing the losses for the separate cases (a, b, etc…), but think about how the loss is a scalar generated by summing the loss for every single case in a trajectory. By minimizing the error in each case by itself, the overall loss will be decreased.
To make it more concrete:
With your example, loss for cases a, b would be -0.105 + 0.105 = 0. Now if we updated the weights w/a gradient step, let’s say:
a) reward=-1, prob=.85, (-log(.85)*reward), loss = -0.162
b) reward=1, prob = .95, (-log(.95)*reward), loss = 0.051
Now the loss (w/our new and improved weights) would be -0.162 + 0.051 = -0.112, which is a lower loss value than 0 above.