Reinforcement learning in environments with large state-action spaces is challenging, as exploration can be highly inefficient. Even if the dynamics are simple, the optimal policy can be combinatorially hard to discover. In this work, we propose a hierarchical approach to structured exploration to improve the sample efficiency of on-policy exploration in large state-action spaces. The key idea is to model a stochastic policy as a hierarchical latent variable model, which can learn low-dimensional structure in the state-action space, and to define exploration by sampling from the low-dimensional latent space. This approach enables lower sample complexity, while preserving policy expressivity. In order to make learning tractable, we derive a joint learning and exploration strategy by combining hierarchical variational inference with actor-critic learning. The benefits of our learning approach are that 1) it is principled, 2) simple to implement, 3) easily scalable to settings with many actions and 4) easily composable with existing deep learning approaches. We demonstrate the effectiveness of our approach on learning a deep centralized multi-agent policy, as multi-agent environments naturally have an exponentially large state-action space. In this setting, the latent hierarchy implements a form of multi-agent coordination during exploration and execution (MACE). We demonstrate empirically that MACE can more efficiently learn optimal policies in challenging multi-agent games with a large number (∼ 20) of agents, compared to conventional baselines. Moreover, we show that our hierarchical structure leads to meaningful agent coordination.
This paper proposes an approach to improve exploration in multiagent reinforcement learning by allowing the policies of the individual agents to be conditioned on an external coordination signal \lambda. In order to find such parametrized policies, the approach combines deep RL with a variational inference approach (ELBO optimization). The paper presents an empirical evaluation, which seems encouraging, but that is also somewhat difficult to interpret given the lack of comparison to other state-of-the-art methods.
Overall, the paper seems interesting, but (in addition to the not completely convincing empirical evaluation), it has two main weaknesses: lack of clarity and grounding in related literature.
=Issues with clarity=
-"This problem has two equivalent solutions". 
This is not so clear; depending on the movement of the preys it might well be that the optimal solution will switch to the other prey in certain cases?
-It is not clear what is really meant with the term "structured exploration". It just seems to mean 'improved'?
-It is not clear that the improvements are due to exploration; my feeling is that is is due to improved statistical strength on a more abstract state feature (which is learned), not unlike:
Geramifard, Alborz, et al. "Online discovery of feature dependencies." Proceedings of the 28th International Conference on Machine Learning (ICML-11). 2011.
However, there is no clear indication that there is an improved exploration policy.
-The problem setting is not quite clear:
The paper first introduces "multi-agent RL", which seems to correspond to a "stochastic game" (also "Markov game"), but then moves on to restrict to the "fully cooperative setting" (which would make it a "Multiagent MDP", Boutilier '96).
It subsequently says it deals only with deterministic problems (which would reduce the problem further to a learning version of a multiagent classical planning problem), but in the experiments do consider stochastically moving preys.
-The paper says the problem is fully observable, but fails to make explicit if this is *individually* fully observable, or jointly. I am assuming the former, but is it not clear how the agents observe this full state in the experimental evaluation.
This is actually a crucial confusion, as it completely changes the interpretation of what the approach does: in the individually observable case, the approach is adding a redundant source of information which is more abstract and thus seems to facilitate faster learning. In the latter case, where agents would have individual observations, it is actually providing the agents with more information.
As such, I would really encourage the authors to better define the task they are considering. E.g., by building on the taxonomies of problems that researchers have developed in the community focusing on decentralized POMDPs, such as:
Goldman, Claudia V., and Shlomo Zilberstein. "Decentralized control of cooperative systems: Categorization and complexity analysis." (2004).
-"Compared to the single-agent RL setting, multi-agent RL poses unique difficulties. A central issue
is the exploration-exploitation trade-off"
That now in particular happens to be a central issue in single agent RL too.
-"Finding the true posteriors P (λ t |s t ) ∝ P (s t |λ t )P (λ t ) is intractable in general"
The paper did not explain how this inference task is required to solve the RL problem.
-In general, I found the technical description impossible to follow, even after carefully looking at the appending. For instance, (also) there the term P (λ |s ) is suddenly introduced without explaining what the term exactly is? Why is the term P(a|λ) not popping up here? That also needs to be optimized, right? I suppose \phi is the parameter vector of the variational approximation, but it is never really stated. The various shorthand notations introduced for clarity do not help at all, but only make the formulas very cryptic.
-The main text is not readable since definitions, e.g., L(Q_r,\tehta,\phi), that are in the appendix are now missing.
-It is not clear to me how the second term of (10) is now estimated?
-"Shared (shared actor-critic): agents share a deterministic hidden layer,"
What kind of layer is this exactly? How does it relate to \lambda ?
-"The key difference is that this model does not sample from the shared hidden layer"
Why would sampling help? Given that we are dealing with a fully observable multiagent MDP, there is no inherent need to randomize at all? (there should be a optimal deterministic joint policy?)
-"There is shared information between the agents"
What information is referred to exactly? 
Also: It is not quite clear if for these domains cloned would be better than completely independent learners (without shared weights)?
-I can't seem to find anywhere what is the actual shape (or type? I am assuming a vector of reals) of the used \lambda.
-in figure 5, rhs, what is being shown exactly? What do the colors mean? Why does there seem to be a \lambda *per* agent now?
=Related work=
I think the paper could/should be hugely improved in this respect. 
The idea of casting MARL as inference has also been considered by:
Learning for Decentralized Control of Multiagent Systems in Large, Partially-Observable Stochastic Environments.
M Liu, C Amato, EP Anesta, JD Griffith, JP How - AAAI, 2016
Stick-breaking policy learning in Dec-POMDPs
M Liu, C Amato, X Liao, L Carin, JP How
International Joint Conference on Artificial Intelligence (IJCAI) 2015
Wu, F.; Zilberstein, S.; and Jennings, N. R. 2013. Monte-carlo
expectation maximization for decentralized POMDPs. In Proc.
of the 23rd Int’l Joint Conf. on Artificial Intelligence (IJCAI-
13).
I do not think that these explicitly make use of a mechanism to coordinate the policies, since they address to true Dec-POMDP setting where each agent only gets its own observations, but in the Dec-POMDP literature, there also is the notion of 'correlation device', which is an additional controller (say corresponding to a dummy agent), which of which the states can be observed by other agents and used to condition their actions on:
Bernstein DS, Hansen EA, Zilberstein S. Bounded policy iteration for decentralized POMDPs. InProceedings of the nineteenth international joint conference on artificial intelligence (IJCAI) 2005 Jun 6 (pp. 52-57).
(and clearly this could be directly included in the aforementioned learning approaches). 
This notion of a correlation device also highlights to potential relation to methods to learn/compute correlated equilibria. E.g.,:
Greenwald A, Hall K, Serrano R. Correlated Q-learning. In ICML 2003 Aug 21 (Vol. 3, pp. 242-249).
A different connection between MARL and inference can be found in:
Zhang, Xinhua and Aberdeen, Douglas and Vishwanathan, S. V. N., "Conditional Random Fields for Multi-agent Reinforcement Learning", in (New York, NY, USA: ACM, 2007), pp. 1143--1150.
The idea of doing something hierarchical of course makes sense, but also here there are a number of related papers:
-putting "hierarchical multiagent" in google scholar finds works by Ghavamzadeh et al., Saira & Mahadevan, etc.
-Victor Lesser has pursued coordination for better exploration with a number of students.
I suppose that Guestrin et al.'s classical paper:
Guestrin, Carlos, Michail Lagoudakis, and Ronald Parr. "Coordinated reinforcement learning." ICML. Vol. 2. 2002.
would deserve a citation, and the MARL field is moving ahead fast, an explanation of the differences with COMA:
Counterfactual Multi-Agent Policy Gradients
J Foerster, G Farquhar, T Afouras, N Nardelli, S Whiteson
AAAI 2018
is probably also warranted.