In multi-agent systems, complex interacting behaviors arise due to heavy correlations among agents. However, prior works on modeling multi-agent interactions from demonstrations have largely been constrained by assuming the independence among policies and their reward structures. In this paper, we cast the multi-agent interactions modeling problem into a multi-agent imitation learning framework with explicit modeling of correlated policies by approximating opponents' policies, which is able to recover agents' policies that can regenerate similar interactions. Consequently, we develop a Decentralized Adversarial Imitation Learning algorithm with Correlated policies (CoDAIL), which allows for decentralized training and execution. Various experiments demonstrate that CoDAIL can better regenerate complex interactions close to the demonstrators and outperforms state-of-the-art multi-agent imitation learning methods.
This paper proposes to model interactions in a multi-agent system by considering correlated policies. In order to do so, the work modifies the GAIL framework to derive a learning objective. Similar to GAIL, the discriminator distinguishes between state, action, next state sequences but crucially the actions here are considered for all agents.
The paper is a natural extension of GAIL/MA-GAIL. I have two major points that need to be addressed.
1. The exposition and significance of some of the theoretical results is unclear.
- The non-correlated and correlated eqns in 2nd and 3rd line in eq. 8 are not equivalent in general, yet connected via an equality.
 In particular, Proposition 2 considers an importance weighting procedure to reweight state, action, next state triplets. It is unclear how this resolves the shortcomings of pi_E^{-1} being inaccessible. Prop 2 shifts from pi_E^{-1} to pi^{-1} and hence, the expectations in Prop 2 and Eq. 11 are not equivalent. 
- More importantly, how are the importance weights estimated in Eq. 12? The numerator requires pi_E^{-1}, which is not accessible. If the numerator and denominator are estimated separately, it becomes a chicken-and-egg problem since the denominator is itself intended to be an imitating the expert policy appearing in the numerator?
2. Missing related work
There is a huge body of missing work in multi-agent interactions modeling and generative modeling. [1, 2] consider modeling of agent interactions via imitation learning and a principled evaluation framework of generalization in the Markov games setting. By sharing parameters, they are also able to model correlations across agent policies and have strong results on generalization to cooperation/competition with unseen agents with similar policies (which wouldn't have been possible if correlations were not modeled). Similarly, [3, 4] are other similar works which consider modeling of other agent interactions/diverse behaviors via imitation style approaches. Finally, the idea of correcting for the mismatch in state, action, next state triplets in Proposition 2 has been considered for model-based off-policy evaluation in [5]. They proposed a likelihood-free method to estimate importance weights, which seems might be necessary for this task as well (re: qs. on how are importance weights estimated?).
Re:experiments. Results look good and convincing for most parts. I don't see much value of the qualitative evaluation in Figure 1. If the KL divergence is low, we can expect the marginals to be better estimated. Trying out various levels of generalization as proposed in [2] would significantly strengthen the paper.
Typos
sec 2.1 Transition dynamics should have range in R+
Proof of Prop 2. \mu instead of u
References:
[1] Learning Policy Representations in Multiagent Systems. ICML 2018.
[2] Evaluating Generalization in Multiagent Systems using Agent-Interaction Graphs. AAMAS 2018.
[3] Machine Theory of Mind. ICML 2018.
[4] Robust imitation of diverse behaviors. NeurIPS 2017.
[5] Bias Correction of Learned Generative Models using Likelihood-free Importance Weighting. NeurIPS 2019.