The paper you mention about multi-agent soft-Q learning is a centralized approach, where each agent are sharing a common critic, with a joint policy (one network giving as output one action per agent). My answer focused on that case.