Multi-Agent Generative Adversarial Imitation Learning
Imitation learning algorithms can be used to learn a policy from expert demonstrations without access to a reward signal. However, most existing approaches are not applicable in multi-agent settings due to the existence of multiple (Nash) equilibria and non-stationary environments. We propose a new framework for multi-agent imitation learning for general Markov games, where we build upon a generalized notion of inverse reinforcement learning. We further introduce a practical multiagent actor-critic algorithm with good empirical performance. Our method can be used to imitate complex behaviors in high-dimensional environments with multiple cooperative or competing agents.
This paper proposes several alternative extensions of GAIL to multi-agent imitation learning settings. The paper includes strong, positive results on a wide range of environments against a suitable selection of baselines. However, insufficient details of the environments is provided to reproduce or fully appreciate the complexity of these environments. If accepted, I would request the authors add these to the appendix and would appreciate details (space permitting) to be discussed in the rebuttal - particularly the state representation.  The more pressing point I would like to raise for discussion in the rebuttal is with regard to the MACK algorithm proposed for the generator. The authors make a justified argument for the novelty of the algorithm, but do not thoroughly justify why they used this algorithm instead of an established MARL algorithm (e.g. Multi-Agent DDPG). It seems that MACK is in itself also a contribution to the literature, but it is not directly compared to a suitable existing baseline. Its use also convolutes whether the gains seen by multi-agent GAIL were due to the different priors used or the different RL algorithm used for the generator. Ideally it would be interesting to see an ablation study showing how multi-agent GAIL performs with the different priors proposed in Figure 1 but with TRPO as in the original paper. Additionally, given the choice to use MACK instead of an established multi-agent RL algorithm, a comparison of multi-agent GAIL with MACK to multi-agent GAIL with multi-agent DDPG would provide empirical evidence of the benefit of MACK.  For future work, I would suggest the authors revisit an assumption of the paper that the number of experts matches directly the number of agents. However, I don't think this assumption is strictly necessary for some of the multi agent extensions proposed and would be an interesting further empirical study as it enables unique use cases where expert data is limited but could generalise across agents or provide partial insight into how some of the agents should act.