Or you can just have
loss = samples_logprob * (reward - baseline)
(logprob is the output of the network, and reward and baseline are leaf variables)
If you do backward, the gradient backpropogated is equivalent to the policy gradient.