Then, I am not sure to understand your question… You want to use the policy as weights for the gradient update ? that way, you should re-write your loss:
weighted_loss = p1*loss(p1) + .. + pn*loss(pn)
weighted_loss.backward()
optimizer.step()