M 3 RL: MIND-AWARE MULTI-AGENT MANAGEMENT REINFORCEMENT LEARNING
Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly learning a policy for each agent to maximize a common reward. In this paper, we aim to address this from a different angle. In particular, we consider scenarios where there are self-interested agents (i.e., worker agents) which have their own minds (preferences, intentions, skills, etc.) and can not be dictated to perform tasks they do not want to do. For achieving optimal coordination among these agents, we train a super agent (i.e., the manager) to manage them by first inferring their minds based on both current and past observations and then initiating contracts to assign suitable tasks to workers and promise to reward them with corresponding bonuses so that they will agree to work together. The objective of the manager is to maximize the overall productivity as well as minimize payments made to the workers for ad-hoc worker teaming. To train the manager, we propose Mind-aware Multi-agent Management Reinforcement Learning (M 3 RL), which consists of agent modeling and policy learning. We have evaluated our approach in two environments, Resource Collection and Crafting, to simulate multi-agent management problems with various task settings and multiple designs for the worker agents. The experimental results have validated the effectiveness of our approach in modeling worker agents' minds online, and in achieving optimal ad-hoc teaming with good generalization and fast adaptation.
This paper studies the problem of coordinating many strategic agents with private valuation to perform a series of common goals. The algorithm designer is a manager who can assign goals to various agents but cannot see their valuation or control them explicitly. The manager has a utility function for various goals and wants to maximize the total revenue. The abstract problem is well-motivated and significant and is an entire branch of study called algorithmic mechanism design. However often many assumptions have to be made to make the problem mathematically tractable. In this paper, the authors take an empirical approach by designing an RL framework that efficiently maximizes rewards across many episodes. Overall I find the problem interesting, well-motivated. The paper is well-written and contains significant experiments to support its point. However, I do not have the necessary background in the related literature to assess the significance of the methods proposed compared to prior work and thus would refrain from making a judgment on the novelty of this paper in terms of methodology. Here are some of my comments/questions to the author on this paper.
(1) I want to clarify how the skills of the agents play a role in the problem setup. Does it show up in the expression for the manager's reward? In particular, does it affect the Indicator for whether a goal is completed Eq. (2) via a process that need not be explicitly modeled but can be observed via a feedback of whether or not the goal is completed? So in the case of resource collection example, the skill set is a binary value for each resource, whether it can be collected or not? 
(2) Related to the first point, the motivation for modeling the agents as maximizing their utility is the assumption that agents do not know their skills. I am wondering, is this really justified? Over the course of episodes, can the agents learn their skills based on the relationship between their intention and the goals they achieve? In the resource collection example, when they reach a resource and are not able to collect it, they understand that they do not have the corresponding skill. Is there a way to extrapolate the results from this paper to such a setting? 
(3) I am slightly concerned about the sample complexity of keeping track of the probability of worker i finishing goal g within t steps with a bonus b. This scales linearly in parameters which usually would be large (such as the number of time-steps). Are there alternate ways to overcome maintaining the UCB explicitly, especially for the number of time-steps? 
Some minor comments on the presentation.
(1) What are the units for rewards in the plots? Is it the average per episode reward? It would be good to mention this in the caption.
(2) There are a few typos in the paper. Some I could catch was,
- Last line in Page 5: "quantitative" -> "quantity"
- Page 8: skills nad preferences -> skills and preferences
- Page 8: For which we combining -> for which we combine