Multi-agent reinforcement learning is a particularly challenging problem. Current methods have made progress on cooperative and competitive environments with particle-based agents using communication and centralized training. Little progress has been made on solutions that could operate in the real world with interaction, dynamics, and humanoid navigation strategies. In this work, we make a significant step in multi-agent models on simulated humanoid navigation by combining Multi-Agent Reinforcement Learning with Hierarchical Reinforcement Learning. Specifically, we develop a partial parameter sharing approach wherein the lower level of the hierarchy is shared enabling learning using decentralized methods. This drastically reduces the overall parameter space in the multi-agent problem and introduces structure in the optimization problem. We build on top of prior foundational work in learning goal-conditioned policies to learn low-level physical controllers for balance and walking. These lower-level controllers are task-agnostic and can be shared by higher-level policies. Overtop of these goal conditioned policies, we can train decentralized heterogeneous policies for multi-agent goal-directed collision avoidance. Surprisingly, our results show that with this combination of methods, RL techniques can be used for finding strong policies. A video of our results on a multi-agent pursuit environment can be seen here 1 .
Summary:
This paper looks at the MARL problem in high-dimensional continuous control settings. To improve learning in this multi-agent setting, they propose to pre-train a lower-level policy that takes as input foot-step goals and is executed for a fixed number of timestep, thereby simplifying both the learning and exploration. 
I'm a bit unsure of how to evaluate this paper. On the one hand, I believe it has several contributions:
- Proposing a new MARL - continuous control environment
- Proposing a new lower-level policy for high-demensional continuous control environments, including how to learn it
- Using it to perform MARL in this environment
On the other hand, it is hard to say what the _main_ contribution is, which in turn makes it difficult to evaluate whether the experimental evaluation is sufficient:
Clearly, a main part of the paper is the work done to construct the hierarchical setup, including goal space, observation space and reward functions. However, this work, as far as I can tell, is separate from the MARL problem. Furthermore, there are several similar ideas already published, so comparison against those (for example by J. Peng, N. Heess or J. Mere) either as argument or even better as experiment, would be helpful to evaluate the quality of the proposed hierarchy.
On the other hand, there is the application of the hierarchical setup to the MARL problem. However, as far as I can tell, there is no difference between applying such a hierarchy to the MARL case and to the single agent problem. Especially if the lower-level component of the hierarchy is pre-trained in a non-MARL setup, it can just be seen as part of the environment from the point of view of the MARL training, offerring limited new insight into MARL. 
I believe in the second paragraph of 4.1 the authors provide some insight into this matter, however, I have to admit I do not understand this paragraph: 
- Why does temporal correlation reduce the non-stationarity of the MARL problem?
- Why does structured exploration reduce the number of network parameters that need to be learned?
- Why does partial parameter sharing make it easier for each agent to estimate other agents potential changes in behavior?
In summary, I think this is interesting work, but a clearer explanation of the relationship between HRL and MARL, as well as a clearer main argument, supported by experimental evidence, would greatly improve this paper.
Edit:
Thank you for your response.
Unfortunately, I don't feel like it sufficiently addresses my questions and concerns. 
I do apologize if my original comment wasn't clear regarding the contribution part of the paper. What I was trying to say is not that I didn't see the individual contributions of the paper, but instead that the paper does multiple things simultaneously, without comparing against the relevant baselines for any of the individual contributions. 
Regarding my questions: I understand where the temporal correlation is coming from in an HRL setting. However, what was not clear to me is how this reduces the non-stationarity of MARL. 
I also understand that HRL can reduce the number of parameters, but I don't see how structured exploration reduces the number of parameters.
And lastly, I also can see how parameter sharing can simplify the learning, but I still don't see how it would allow agents to estimate the behaviour change of other agents easier. 
I feel like in the paragraph in questions, a lot of causes and effects are mixed up and more careful descriptions of the benefits of the algorithm would help. 
I want to re-iterate that I think that the submitted work by the authors is impressive and can provide valuable insights, but I believe it requires more work and more relevant baselines.