In multiagent systems (MASs), each agent makes individual decisions but all of them contribute globally to the system evolution. Learning in MASs is difficult since each agent's selection of actions must take place in the presence of other co-learning agents. Moreover, the environmental stochasticity and uncertainties increase exponentially with the increase in the number of agents. Previous works borrow various multiagent coordination mechanisms into deep learning architecture to facilitate multiagent coordination. However, none of them explicitly consider action semantics between agents that different actions have different influence on other agents. In this paper, we propose a novel network architecture, named Action Semantics Network (ASN), that explicitly represents such action semantics between agents. ASN characterizes different actions' influence on other agents using neural networks based on the action semantics between them. ASN can be easily combined with existing deep reinforcement learning (DRL) algorithms to boost their performance. Experimental results on StarCraft II micromanagement and Neural MMO show ASN significantly improves the performance of state-of-the-art DRL approaches compared with several network architectures.
This paper proposes a neural network architecture that provides an agent-agent based embeddings that are used for actions that directly affect specific agent. Proposed architectural choice exploits (implicitly assumed) independence of some actions wrt. observations of agents that it is not directly affecting. Authors perform experiments in minigame of SC2 (controlling small armies) and in NeuralMMO (analogous setup).
It is worth stressing, that apart from conducting typical analysis, authors also put forward more qualitative hypotheses, that are then verified by custom experiments. I highly value this way of conducting research, as it provides way more insights into the method than just pure return plots, often encountered in similar papers.
Major comments
- Please be more explicit in the abstract, when talking about the experiments conducted. Claiming "Experimental results on StarCraft II and Neural MMO show ASN significantly improves the performance of state-of-the-art DRL approaches compared with several network architectures." could be misinterpreted as providing a solution that can play SC2 better than state-of-the-art methods, while the actual result is to be able to micromanage small fights in this environment well, and thus it would be fairer to say "on StarCraft II Minigame" or more explicitely "StarCraft II small scale fight simulation" or anything else in this spirit. These are great research domains, there is no need to overclaim the results.
- A proposed architecture assumes (to some extent) an independence of effects of action-agent pairs (action aimed at player i is conditionally independent from observation of agent j). There is still a way to represent it, but only by bypassing the proposed architecture, and expressing this behaviour directly in e^i. While this sort of decomposition makes sense for various heavily localised problems (such as micro-management of fights in games like SC2 or NeuroMMO) it looks like a potential limitation, and assumption, that is not articulated in the text. Have authors performed any analysis looking into this phenomenon?
- It is unclear, how ASN compares or relates to the architecture used in OpenAI Five network (https://d4mucfpksywv.cloudfront.net/research-covers/openai-five/network-architecture.pdf) which arguably should be used as a baseline, given that it follows a similar idea of producing per-agent embedding, that influences part of action space, that is directly related to influencing other agents (targeting opponents). Despite not being peer reviewed, it is a well known, and clearly described method, which should at the very least be acknowledged as prior work.
- Some equations are hard to parse, e.g.:
-- page 5, sum_i=1^b [(y_t^tot - Q_tot(s,a; theta))^2]  <- "i" is never used in the equation (I guess it changed meaning, and is no longer agent identity, but rather batch index?)
-- page 4, what does expectation over "t" mean? In the introduced notation, expectation is taken wrt. control policy pi_i, interpreted as a joint probability distribution over agent actions and environment state transition. Can this be unified?
Minor comments
- "When one of the arm units dies" -> "When one of the army units dies"
- details of ASN-PPO could be safely moved to the Appendix, as it is a direct substitution regular policy updates with (2), rather than a separate contribution that requires in-depth description (and given that paper is quite long, it could use a bit of message "sharpening")
I am happy to revisit (and increase) the rating assigned, once authors address the comments above, especially the relation to OpenAI Five architecture.