@confused, I believe that I can help explain the math a bit.
Notice that in this line 2, the rewards from an episode are standardized to be unit normal, meaning the distribution of the standardized rewards will have mean 0 and a variance of 1. This serves as a measure of the “goodness” of an action, relative to the others in the trajectory. Standardized rewards > 0 means that we want to increase the probability of taking those actions given the state, and the reverse for rewards < 0.
Let’s look at the case for the highest loss, which is when there is a very low probability of taking an action that led to a very high reward. Try plugging that into the loss: (-log_prob * reward) term, and you’ll see that it results in a very high loss, which we want to minimize.
I hope I was able to help!