Concerning the soft-Q learning approach, the adaptation to discret worlds looks simple:
in the critic update, use
Q(a,s) = r(a,s) + sum_s’ ( T(s’|a,s) * V(s’) )
V(s) = log( sum_a exp( Q(a,s) / alpha )
and directly compute the new policy
pi(a|s) = softmax( Q / alpha ) (a,s)
directly for all agents.