You should look into the REINFORCE algorithm for that.
But from what I remember, the gist is that the reward should be the expected reward by following the probs that you computed.
If you take the argmax, you don’t have the expected reward of your computed probs and so you don’t have an unbiased estimate of the gradients.