There is no need to learn which action is or is not allowed. By setting the forbidden actions probabilities to zero, your agent will only explore the allowed ones and learn what is the best action out of the allowed set.
However, if you want to reduce the log_prob of the forbidden actions in your gradient step, you can add a gradient direction that simply reduce the log_probabilities for the forbidden actions. in other words, instead of:
theta = theta - alpha * grad_logprobs * returns # (REINFORCE iteration)

you do:
theta = theta - alpha * (grad_logprobs * returns + grad_logprobs * forbidden) 
# where forbidden = 0 if a state-action is allowed

or again:
theta = theta - alpha * grad_logprobs * (returns+forbidden)

But I think it would be longer to train, as you will have to explore a bigger space.