Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
We explore deep reinforcement learning methods for multi-agent domains. We begin by analyzing the difficulty of traditional algorithms in the multi-agent case: Q-learning is challenged by an inherent non-stationarity of the environment, while policy gradient suffers from a variance that increases as the number of agents grows. We then present an adaptation of actor-critic methods that considers action policies of other agents and is able to successfully learn policies that require complex multiagent coordination. Additionally, we introduce a training regimen utilizing an ensemble of policies for each agent that leads to more robust multi-agent policies. We show the strength of our approach compared to existing methods in cooperative as well as competitive scenarios, where agent populations are able to discover various physical and informational coordination strategies.
Summary
-----------------
The paper presents a novel actor-critic algorithm, named MADDPG, for both cooperative and competitive multiagent problems. MADDPG relies on a number of key ideas:
	1) The action value functions are learned in a 'centralized' manner, meaning that it takes into account the actions of all other players. This allows to evaluate the effect of the joint policy on each agents long term reward. To remove the need of knowing other agents' actions, the authors suggest that each agent could learn an approximate model of their policies.
	2) The learned policies can only use local information, so that they can be used by each agent without relying on further communication.
	3) It learns a collection of policies for each agent, rather than just one. At each episode during the learning process, each agent draws uniformaly a policy from its ensemble. Sampling a policy from an ensemble reduces the non-statinoarity caused by multiple agents learning at the same time.
Several interesting, both cooperative and competitive, are considered to evaluate the proposed algorithm. Simulation results show the benefit of using ensembles and improvement with respect to independent DDPG agents.
Major comments
-----------------
The main algorithmic contribution of the paper seems to be the policy ensemble to surmount nonstationarity. This is an interesting idea. However, very little insight is provided. Do the authors have any hypothesis of why this seems to work well? 
The environments considered in Sec. 5 constitute an interesting benchmark. However, the proposed MADDPG algorithm is compared against independent DDPG agents. Although this validates the effectiveness of the algorithm, it would be more interseting to compare with centralized DDPG, as well as with other centralized critic approaches (e.g., [6]), and other ensemble RL algorithms.
The organization of the paper is somehow unbalanced. For instance, it reviews well known methods, while there are some important points that remain unclear. In particular, it seems that the algorithm learns as many action value functions as agents, so that it learns a parameter theta_i per agent. However, in lines 333-334, the authors mention that the complexity grows linearly for some Q (without subscript), rather than considering the computational burden of learning a set of action value functions {Q_i}. 
Minor comments
-----------------
- Sentence in lines 47-48 is difficult to understand.
- The use of bold font notation seems arbitrary. Why are { pi_i } bold while { theta_i } aren't ?
- Word 'Reference' should precede '[11]' in line 81.