Then this could be a problem. The idea of reinforcement learning is that you use the gradient path of your predictions (in your case the log_probs) to propagate the reward which was multiplied with the predictions. If you don’t have the gradient path for the predictions (as ist seems to be the case in your code snippets) you cannot successfully propagate gradients through the network?
Have you monitored the model’s parameters using plain SGD for optimization? Do they change?