I am trying to understand: you want to learn a policy composed by N (supposedly) independent multinomial distributions.
So, something like  \pi(a_i = k | s) ~ p_i^k (1-p_i)^k up to a normalization for i=1…N
And you want to learn it as a single distribution: \pi(A=[k1..kn] | s) right?
In that case, assuming all distributions are independent, you can sum the log_prob of all distributions and back propagate this sum (times a reward). If all parameters are given to the optimizer, it should work.