Yes, it should be negative, but the learning rate positive. I was too fast on this. The correct iteration is:
theta = theta + alpha * grad_logprobs * (returns - forbidden)

That way, you optimize the return, but still minimize the log_prob of forbidden actions.
By the way, you can do this and still force your agent to only explorate allowed actions, so you don’t lose time with exploration.