We explore a collaborative multi-agent reinforcement learning setting where a team of agents attempts to solve cooperative tasks in partially-observable environments. In this scenario, learning an effective communication protocol is key. We propose a communication architecture that allows for targeted communication, where agents learn both what messages to send and who to send them to, solely from downstream task-specific reward without any communication supervision. Additionally, we introduce a multi-stage communication approach where the agents co-ordinate via multiple rounds of communication before taking actions in the environment. We evaluate our approach on a diverse set of cooperative multi-agent tasks, of varying difficulties, with varying number of agents, in a variety of environments ranging from 2D grid layouts of shapes and simulated traffic junctions to complex 3D indoor environments. We demonstrate the benefits of targeted as well as multi-stage communication. Moreover, we show that the targeted communication strategies learned by agents are both interpretable and intuitive.
The authors present a study on multi-agent communication.
Specifically, they adapt communication to be targeted and multi-staged.
Experiments on  2 synthetic datasets and 1 3D visual dataset confirm that both additions are beneficial
Overall, this paper was somewhat clear and more importantly includes experiments on House3D, a more realistic dataset.
My main concern is the following: the method is not about targeting, but about selectively hearing.
If agents are sharing the reward then why should targeted communication be beneficial at all? Isn't the optimal strategy to just communicate everything to everyone? I understand that they should be selective at the listening side to properly integrate only the relevant information (so, attend over all received messages), but why should we expect the speaker to apriori know who this message should go to? Moreover, I don't really understand how targeted communication can even work (in the way the authors explain it) since the agents have partial information (e.g., in shapes they only see 5x5 around them), so they don't really know who is where --  but I could potentially see this working should the agents put information about their own identity and location.  So, given the positive results that the authors get, my understanding is that the signature doesn't have information about who should the recipient of the information be but more about what where the properties of the sender of this information.  So, based on my understanding, I don't feel that the flow of the story quite matches what is really happening and this might be very confusing for prospective readers. Can the authors elaborate on this, aim i getting things wrong?
There is literally no information about model size (or at least I wasn't able to find any). Is there any weight-sharing across agents? Do you obtain CommNets by using the implementations of the authors or by ablating the signature-part of your model? Moreover, why do agents have a limited view window on the SHAPES -- is (targeted) communication redundant when agents have full observability? The part about how multi-staged communication is implemented is quite cryptic at the moment -- is multi-staged the fact that the message is out-putted by processing with a recurrent unit? The messages is factorized into two parts k and u leading to a vector of size D -- what happens should we have one message of size D (rather than factorizing into 2), something like this would control for any improvements obtained from increases the parameters of the model.
Finally,  if the premises of the paper is to define more effective communication protocols, evident in the use of continuous communication, (rather than studying what form can multi-agent communication etc etc), a necessary baseline  (especially in cases where agents share reward), is to communicate the full observation (rather than a function of it).  This baseline is not presented here and it's absolutely necessary.