Learning to cooperate is crucially important in multi-agent environments. The key is to understand the mutual interplay between agents. However, multi-agent environments are highly dynamic, where agents keep moving and their neighbors change quickly. This makes it hard to learn abstract representations of mutual interplay between agents. To tackle these difficulties, we propose graph convolutional reinforcement learning, where graph convolution adapts to the dynamics of the underlying graph of the multi-agent environment, and relation kernels capture the interplay between agents by their relation representations. Latent features produced by convolutional layers from gradually increased receptive fields are exploited to learn cooperation, and cooperation is further improved by temporal relation regularization for consistency. Empirically, we show that our method substantially outperforms existing methods in a variety of cooperative scenarios.
The paper addresses the problem of coordination in the multi-agent reinforcement learning setting. It proposes the value function factorization similar to independent Q-learning conditioning on the output of the graph convolutional neural network, where the graph topology is based on the agents’ nearest neighbours. The paper is interesting and has some great ideas, for example, KL term to ensure temporal cooperation consistency. However, the paper has its drawbacks and I feel obliged to point them out below. I vote for the weak acceptance of this paper.
One of the main drawbacks of the paper is that it is extremely hard to grasp. Even the Abstract and Introduction are hard to understand without having a pass over the whole paper. The authors often use vague terms such as 'highly dynamic environments' or 'dynamics of the graph' which make it hard to understand what they mean. The paper would benefit from a more precise language. Some of the important notions of the paper are used before they are introduced, which make the general picture very hard to understand and to relate the work to the existing research.
'Related Work' section seems to be missing some recent work applying graph neural networks to multi-agent learning settings:
• Malysheva, Aleksandra, Tegg Taekyong Sung, Chae-Bong Sohn, Daniel Kudenko, and Aleksei Shpilman. "Deep Multi-Agent Reinforcement Learning with Relevance Graphs." arXiv preprint arXiv:1811.12557 (2018).
• Agarwal, Akshat, Sumit Kumar, and Katia Sycara. "Learning Transferable Cooperative Behavior in Multi-Agent Teams." arXiv preprint arXiv:1906.01202 (2019).
My questions to the authors:
• In section 3 you mention 'a set of neighbours ..., which is determined by distance or other metrics'. Can you elaborate on that? What are these metrics in your case?
• Just before the Section 3.1, you say 'In addition, in many multi-agent environments, it may be costly and less helpful to take all other agents into consideration.' Have you run any experiments on that? In the appendix, you show, that making the neighbourhood smaller negatively affects the performance, but what if you make it bigger? Ideally, I would like to see an extended version of Figure 8 and 9 in the main part of the paper since they are very interesting and important for the claims the paper makes.
• At the end of Section 3.1, you mention the soft update of the target network. Later, in 3.3, you say that that you do not use the target network. Can you elaborate more on that?
• In Equation 4, is it a separate KL for each of the attention heads? If yes, this is not clear from the formula.
• It will be useful to see the ablation experiments for all of the testbeds, not only for Battle.
• Why do you think the DQN performance drops in the second half of the training in Figure 4 for all of the runs?
• Have you tried summation instead of the mean aggregation step?
I will put comments for particular parts of the paper below.
ABSTRACT
>>> ...environments are highly dynamic
What do you mean precisely here?
>>> ...graph convolution adapts to the dynamics of the underlying graph of the multi-agent environment
What is the 'dynamics of the underlying graph'? What is the 'graph of the multi-agent environment'?
>>> 'coordination is further boosted'
Not sure that 'boosted' is the right word here.
INTRODUCTION
>>> '...where mutual interplay between humans is abstracted by their relations'
Not sure what it means.
>>> we consider the underlying graph of agents...
The agent graph has not been introduced yet.
>>> DGN shares weights among all agent(s) making it easy to scale
What do you mean precisely by 'easy to scale'? Can you support this claim?
>>> We empirically show the learning effectiveness of DGN in jungle
Needs a reference to the testbed.
>>>  ... interplay between agents and abstract relation representation
What is 'abstract relation representation?
>>> We consider partially observable environments.
What do you mean precisely by that? What is the MDP formalism most suitable for your problem statement? What is objective under your formalism?
>>> However, more convolutional layers will not increase the local region of node i.
What do you mean by that?
>>> As the number and position of agents vary over time, the underlying graph continuously changes, which brings difficulties to graph convolution.
What kind of difficulties?
>>> As the action of agent can change the graph at next timestep which makes it hard to learn Q function.
Why does it make it hard?
>>> DGN can also be seen as a factorization of a centralized policy that outputs actions for all the agents to optimize the average expected return.
It would be useful for the reader to compare your approach with all the others type of the value function factorization. To me, your approach looks like a more sophisticated version of independent Q-learning, is that true?
Minor comments:
* In 3.2 it would be very helpful to put the dimensions for all of the variables for easier understanding.
* The brackets in the equation 3 are not very clear (what are you concatenating across?)
* In section 4, when describing an environment you say 'local observation that contains a square view with 11x11 grids'. What is the total size of the environment?
* The performance plots for Battle include ablations before the ablation subsection is introduced. This is a bit confusing.
* All figures/tables captions should be more detailed and descriptive.
* ‘However, causal influence is not directly related to the reward of environment.’ Should be ‘of the environment’.