Usually, the probabilities are obtained from policy_network as a result of a softmax operation.

you should put a LogSoftmax in the policy network’s last layer, not Softmax, precisely for the reason you described.