The underlying logic is that you want to maximize the sum of the rewards under the policy, which can be obtained by a gradient descent with the derivative of this sum. And the derivative of this sum is the sum of R*derivative(log(policy)).
That’s the policy-gradient theorem, and it’s easy to prove (using the fact that d(pi) = d(log(pi))*pi )
Now, how to interpret the idea in a less formal way: you want to maximize the log-likelihood where rewards are height and minimalize it when rewards are small. Of course, there is a singularity when the policy is deterministic (p(Si) = Ai with probability 1), but such a case is never reached in a sub-optimal way if the policy is not initialized in that singular way. Usually the policy is initialized with a small normal noise around a uniform distribution.