We consider tackling a single-agent RL problem by distributing it to n learners. These learners, called advisors, endeavour to solve the problem from a different focus. Their advice, taking the form of action values, is then communicated to an aggregator, which is in control of the system. We show that the local planning method for the advisors is critical and that none of the ones found in the literature is flawless: the egocentric planning overestimates values of states where the other advisors disagree, and the agnostic planning is inefficient around danger zones. We introduce a novel approach called empathic and discuss its theoretical aspects. We empirically examine and validate our theoretical findings on a fruit collection task.
Summary
The paper is well-written but does not make deep technical contributions and does not present a comprehensive evaluation or highly insightful empirical results.
Abstract / Intro
I get the entire focus of the paper is some variant of Pac-Man which has received attention in the RL literature for Atari games, but for the most part the impressive advances of previous Atari/RL papers are in the setting that the raw video is provided as input, which is much different than solving the underlying clean mathematically abstracted problem (as a grid world with obstacles) as done here and evident in the videos.  Further it is honestly hard for me to be strongly motivated about a paper that focuses on the need to decompose Pac-man into sub-agents/advisor value functions.
Section 2
Another historically well-cited paper for MDP decomposition:
  Flexible Decomposition Algorithms for Weakly Coupled Markov Decision Problems, Ronald Parr. UAI 98.
  https://dslpitt.org/uai/papers/98/p422-parr.pdf
Section 3
Is the additive reward decomposition a required part of the problem specification?  It seems so, i.e., there is no obvious method for automatically decomposing a monolithic reward function over advisors.
Section 4
* Egocentric:
Definition 1: Sure, the problem will have local optima (attractors) when decomposed suboptimally -- I'm not sure what new insight we've gained from this analysis... it is a general problem with any function approximation scheme that does not guarantee that the rank ordering of actions for a state is preserved.
* Agnostic
Other than approximating some type of myopic rollout, I really don't see why this approach would be reasonable?  I am surprised it works at all though my guess is that this could simply be an artifact of evaluating on a single domain with a specific structure.
* Empathic
This appears to be the key contribution though related work certainly infringes on its novelty.  Is this paper then an empirical evaluation of previous methods in a single Pac-man grid world variant?
I wonder if the theory of DEC-MDPs would have any relevance for novel analysis here?
Section 5
I'm disappointed that the authors only evaluate on a single domain; presumably the empathic approach has applications beyond Pac-Man?
The fact that empathic generally performs better is not at all surprising.  The fact that a modified discount factor for egocentric can also perform well is not surprising given that lower discount factors have often been shown to improve approximated MDP solutions, e.g.,
  Biasing Approximate Dynamic Programming with a Lower Discount Factor
  Marek Petrik, Bruno Scherrer (NIPS-08).	
  http://marek.petrik.us/pub/Petrik2009a.pdf
***
Side note:
The following part is somewhat orthogonal to the review above in that I would not expect the authors to address this on revision, *but* at the same time I think it provides a connection to the special case of concurrent action decomposition into advisors, which could potentially provide a high impact direction of application for this work (i.e., concurrent problems are hard and show up in numerous operations research problems covering inventory control, logistics, epidemic response).
For the special case that each advisor is assigned to one action in a factored space of concurrent actions, the egocentric algorithm would be very close to the Hindsight approximation in Section 6 of this paper (including an additive decomposition of rewards):
  Planning in Factored Action Spaces with Symbolic Dynamic Programming
  Aswin Nadamuni Raghavan, Alan Fern, Prasad Tadepalli, Roni Khardon, and Saket Joshi (AAAI-12).
  https://www.aaai.org/ocs/index.php/AAAI/AAAI12/paper/download/5012/5336
This simple algorithm is hard to beat for the following reason that connects some details of your egocentric and empathic settings: rather than decomposing a concurrent MDP into independent problems per concurrent action, the optimization of each action (by each advisor) is done in sequence (advisors are ordered) and gets to condition on the previously selected advisor actions.  So it provides an alternate paradigm where advisors actually get to see and condition their policy on what other advisors are doing.  In my own work comparing optimal concurrent solutions to this approach, I have found this approach to be near-optimal and much more efficient to solve since it exploits decomposition.
Why is this relevant to this work?  Because (a) it suggests another variant of the advisor decomposition that at least makes sense in the case of concurrent actions (and perhaps shared actions though this would require some extension) and (b) it suggests there are more options than just the full egocentric and empathic settings in this important class of concurrent action problems that are necessarily solved in practice for large action spaces by some form of decomposition.  This could be an interesting direction for future exploration of the ideas in this work, where there might be additional technical novelty and more space for empirical contributions and observations.