Learning Attentional Communication for Multi-Agent Cooperation
Communication could potentially be an effective way for multi-agent cooperation. However, information sharing among all agents or in predefined communication architectures that existing methods adopt can be problematic. When there is a large number of agents, agents cannot differentiate valuable information that helps cooperative decision making from globally shared information. Therefore, communication barely helps, and could even impair the learning of multi-agent cooperation. Predefined communication architectures, on the other hand, restrict communication among agents and thus restrain potential cooperation. To tackle these difficulties, in this paper, we propose an attentional communication model that learns when communication is needed and how to integrate shared information for cooperative decision making. Our model leads to efficient and effective communication for large-scale multi-agent cooperation. Empirically, we show the strength of our model in a variety of cooperative scenarios, where agents are able to develop more coordinated and sophisticated strategies than existing methods.
Summary: This paper describes and evaluates a complex reinforcement learning algorithm for achieving cooperation within a team of agents.  As I understand it, in this design, the agents all share the same policy network, but they sense different things.  To help the agents coordinate their behavior, the system learns when to communicate and combine “thoughts” (outputs from a layer of a neural network) with each other.  The authors empirically analyze the system in three different domains (cooperative navigation, cooperative push ball, and Predator-Prey).  The results achieved appear to outperform existing algorithms designed for similar situations.  Quality: The authors appear to combine together state-of-the-art algorithmic mechanisms in a very clever way.  The results suggest that the algorithm is effective.  I very much like that the authors evaluated the system in three different domains.  These attributes make the paper quite appealing.  But no paper is perfect (at least from the eye of reviewers).  In my opinion, the quality of the paper could be improved in three ways:  1.  Provide more details in order to make the paper reproducible: To me, the algorithm and results do not appear to be reproducible given the descriptions provided in the paper.  While the paper does a good job of overviewing the system, important details (parameter values, network designs, etc.) that are needed to reproduce the results are not given (at least that I saw).  I believe that it is likely that someone else trying to implement similar mechanisms but without the same details would get very different results (particularly if there were not experts in the field), as I believe these details are likely critical to the success of the system.  That said, I realize that providing sufficient details to reproduce complex algorithms is very difficult in a short paper format.  2.  More thorough algorithmic analysis: From the analysis of the results, it is a bit difficult to determine what makes the system tick.  Comparisons to previously developed algorithms are nice to establish baselines, but do not thoroughly isolate in all instances what makes the algorithms tick.  The authors do add nice analysis (e.g., testing ATOC with and without communication).  But even without communication, ATOC outperforms the other algorithms substantially in cooperative pushball (Table 2).  Is this because the learning mechanisms used in ATOC are just better, or did the authors just take more time to refine and optimize ATOC for the given domains more than the other algorithms?  3.  Statistical comparisons: It is also somewhat difficult to interpret the differences between the various algorithms, as no statistical tests or error bars are provided to communicate to the reader significance.  Clarity: The organization of the paper is quite good, and it is definitely possible for a reader to get a good high-level understanding of what is being done.  I do have a few suggestions (nitpicks in some sense) that may be helpful in providing additional clarity.  1.  At least from my perspective, the abstract and introduction didn’t spell out the domain very well up front.  Agents that “cooperate” with each other can take on many forms.  The algorithmic architecture described herein addresses one of these forms, that of a team of computer agents that share many aspects of the same “brain.”.  As recommended by Shoham et. al, 2007 (If multi-learning is the answer, what is the question), I recommend better spelling this out from the onset.  The domain addressed in this work (and hence potentially the kinds of algorithms that will work) is very different from other recent and older work in which communication of AI systems is designed and evaluated, including systems that interact with people and systems in which cooperation can be beneficial but in which the agents do not share the same interests.    2.  I may have missed it, but a good description of how the rewards are computed for each domain appears to be missing.  For example, I’m not sure how good a reward of -0.04 is (or what it represents) in the “cooperative navigation” task (Table 1).  3.  Several times throughout the paper, the authors use the terms “barely help,” “barely adapt,” and “hardly adapts” to describe other algorithms.  Each time these terms are used, I got confused.  The language seemed to be imprecise and could be interpreted as just attempts to “beat up” previous work.  I believe the authors are trying to communicate that the previous algorithms can sometimes have issues “scaling up” because of various design decisions — I just recommend using different language to describe that.  Originality: The work builds on and appears to improve upon a line of work that is emerging in the recent literature:  communication in team settings in which a single designer controls all robots.  I believe that the algorithmic architecture is novel.  Significance: On one hand, the authors appear to be combining together advanced methods in a very clever way.  The results achieved are relatively good, and it does appear that the work advances the state-of-the-art.  On the other hand, the perspective is a little bit strange, neither using a true multi-agent system (wherein all agents are separate) nor a fully centralized system.  Since the designer appears to have full control of the algorithms used by all robots, is there a reason that a computationally adept, but fully centralized system could not be used for computation rather than a “multi-agent system.”  Overall, the assumptions of the paper are not fully spelled out, nor are the algorithmic mechanisms thoroughly evaluated in isolation.  Too me, this dampens the significance of the contributions, but certainly does not eliminate them.  I find this paper to be a nice work.