Learning Multiagent Communication with Backpropagation
Many tasks in AI require the collaboration of multiple agents. Typically, the communication protocol between agents is manually specified and not altered during training. In this paper we explore a simple neural model, called CommNet, that uses continuous communication for fully cooperative tasks. The model consists of multiple agents and the communication between them is learned alongside their policy. We apply this model to a diverse set of tasks, demonstrating the ability of the agents to learn to communicate amongst themselves, yielding improved performance over non-communicative agents and baselines. In some cases, it is possible to interpret the language devised by the agents, revealing simple but effective strategies for solving the task at hand.
First, we simplify and extend the graph neural network is architecture can be used to control groups of cooperating f multilayer neural networks f i that take as input vectors odel takes as input a set of vectors {h 
} (we will call K the number of hops in the network). output them directly, so that the model outputs a vector an feed them into another network to get a single vector or by a nonlinearity :
d, and the matrix can be dynamically sized because the oordinate. 
(we will call K the number of hops in the network). tput them directly, so that the model outputs a vector feed them into another network to get a single vector or y a nonlinearity :
, and the matrix can be dynamically sized because the ordinate.
tion Processing Systems (NIPS 2016). Do not distribute.
uthor(s) n t implify and extend the graph neural network re can be used to control groups of cooperating r neural networks f i that take as input vectors as input a set of vectors {h l call K the number of hops in the network).
directly, so that the model outputs a vector into another network to get a single vector or nearity :
matrix can be dynamically sized because the ssing Systems (NIPS 2016). Do not distribute.
ically sized, and the matrix can be dynamically sized because the r than by coordinate.
eural Information Processing Systems (NIPS 2016). Do not distribute.
The paper deals with centralized learning in the case of multi-agent collaboration problems. More specifically, it proposes a deep-learning approach for learning to collaborate when every agent needs to communicate with others (each agent only gets a partial view of the environment), but no communication protocol has been stated beforehand.  
The model is a deep network which consists of a stack of layers, with parameter sharing between modules of a same layer. This parameter sharing allows the number of agents to vary during the task. Also, it allows to drastically reduce the number of parameters to be learned. The key idea of the paper is to use the output of every module of a given layer to build the communication input for the next layer. While this appears to obtain interesting results in the reported experiments, I find this proposal very straightforward and poorly innovative, as it corresponds to a quiet classical neural network structure. In fact, that is a classical network with shared parameters, except that the output of any agent $i$ is concatenated  with (rather than added to) the summed outputs of every others to form the input of this agent $i$ for the next layer. I am not sure about the intuition of what this could bring in the context of the considered tasks.   Experiments should have at least compared the proposal with a classical network with parameter sharing to highlight the benefits of the proposed mechanism. Also, a centralized controller that has access to all observations of all agents should perform better than the studied setting. So, if the dense baseline corresponds to such a centralized controller scheme (as suggested by the authors answer), there is no reason for me why this baseline does not get greatly better results... Since getting the whole information, this baseline should perform better than approaches with communication between partial views. Maybe that is because it has too many parameters and this is due to an optimization problem ? That's also why I asked a baseline as dense but with shared parameters between modules of a same layer to be able to compare with the proposal and to lower the learning complexity. Anyway, I am not sure on how the dense baseline works in the various settings...    Other comments:    - As the main argument is to maintain a network where the structure can be dynamically sized, it would have been useful to give in introduction some examples of applications where this dynamicity is particularly important    - I am not sure about the relevance of presenting the reinforcement policy gradient in section 2, as : 1) the proposed model can be learned via supervised learning in some applications and 2) this is not at the heart of the proposal at all. I would give it (with more details to be more self-contained) after the presentation of the model (maybe in a section Learning or something like that). Like this it confuses the discourse a lot from my point of view    - Not enough details of the experimental settings are given in the main paper. Very important points such as inputs given to the agents are given in the supplementary material. This complicates the analysis of the results    - Not enough explanations / intuitions are given w.r.t. what agents communicate for the different tasks. Wouldn't it be possible to give a simple example of what is learned for instance in a tiny lever pulling instance ? The section 5.2.2 does not help a lot to understand what is exchanged between agents