Many tasks in artificial intelligence require the collaboration of multiple agents. We exam deep reinforcement learning for multi-agent domains. Recent research efforts often take the form of two seemingly conflicting perspectives, the decentralized perspective, where each agent is supposed to have its own controller; and the centralized perspective, where one assumes there is a larger model controlling all agents. In this regard, we revisit the idea of the master-slave architecture by incorporating both perspectives within one framework. Such a hierarchical structure naturally leverages advantages from one another. The idea of combining both perspectives is intuitive and can be well motivated from many real world systems, however, out of a variety of possible realizations, we highlights three key ingredients, i.e. composed action representation, learnable communication and independent reasoning. With network designs to facilitate these explicitly, our proposal consistently outperforms latest competing methods both in synthetic experiments and when applied to challenging StarCraft 1 micromanagement tasks.
This paper investigates multiagent reinforcement learning  making used of a "master slave" architecture (MSA). On the positive side, the paper is mostly well-written, seems technically correct, and there are some results that indicate that the MSA is working quite well on relatively complex tasks. On the negative side, there seems to be relatively limited novelty: we can think of MSA as one particular communication (i.e, star) configuration one could use is a multiagent system. One aspect does does strike me as novel is the "gated composition module", which allows differentiation of messages to other agents based on the receivers internal state. (So, the *interpretation* of the message is learned). I like this idea, however, the results are mixed, and the explanation given is plausible, but far from a clearly demonstrated answer.
There are some important issues that need clarification:
* "Sukhbaatar et al. (2016) proposed the “CommNet”, where broadcasting communication channel among all agents were set up to share a global information which is the summation of all individual agents. [...] however the summed global signal is hand crafted information and does not facilitate an independently reasoning master agent."
-Please explain what is meant here by 'hand crafted information', my understanding is that the f^i in figure 1 of that paper are learned modules?
-Please explain what would be the differences with CommNet with 1 extra agent that takes in the same information as your 'master'.
*This relates also to this: 
"Later we empirically verify that, even when the overall in-
formation revealed does not increase per se, an independent master agent tend to absorb the same
information within a big picture and effectively helps to make decision in a global manner. Therefore
compared with pure in-between-agent communications, MS-MARL is more efficient in reasoning
and planning once trained. [...] 
Specifically, we compare the performance among the CommNet model, our
MS-MARL model without explicit master state (e.g. the occupancy map of controlled agents in this
case), and our full model with an explicit occupancy map as a state to the master agent. As shown in
Figure 7 (a)(b), by only allowed an independently thinking master agent and communication among
agents, our model already outperforms the plain CommNet model which only supports broadcast-
ing communication of the sum of the signals."
-Minor: I think that the statement "which only supports broadcast-ing communication of the sum of the signals" is not quite fair: surely they have used a 1-channel communication structure, but it would be easy to generalize that.
-Major: When I look at figure 4D, I see that the proposed approach *also* only provides the master with the sum (or really mean) with of the individual messages...? So it is not quite clear to me what explains the difference.
*In 4.4, it is not quite clear exactly how the figure of master and slave actions is created. This seems to suggest that the only thing that the master can communicate is action information? It this the case?
* In table 2, it is not clear how significant these differences are. What are the standard errors?
* The section 3.2 explains standard things (policy gradient), but the details are a bit unclear. In particular, I do not see how the Gaussian/softmax layers are integrated; they do not seem to appear in figure 4?
* I cannot understand figure 7 without more explanation. (The background is all black - did something go wrong with the pdf?)

Details:
* references are wrongly formatted throughout. 
* "In this regard, we are among the first to combine both the centralized perspective and the decentralized perspective"
This is a weak statement (E.g., I suppose that in the greater scheme of things all of us will be amongst the first people that have walked this earth...)
* "Therefore they tend to work more like a commentator analyzing and criticizing the play, rather than
a coach coaching the game."
-This sounds somewhat vague. Can it be made crisper?
* "Note here that, although we explicitly input an occupancy map to the master agent, the actual infor-
mation of the whole system remains the same."
This is a somewhat peculiar statement. Clearly, the distribution of information over the agents is crucial. For more insights on this one could refer to the literature on decentralized POMDPs.