In recent years, reinforcement learning methods have been applied to model gameplay with great success, achieving super-human performance in various environments, such as Atari, Go and Poker. However, those studies mostly focus on winning the game and have largely ignored the rich and complex human motivations, which are essential for understanding the agents' diverse behavior. In this paper, we present a multi-motivation behavior modeling which investigates the multifaceted human motivations and models the underlying value structure of the agents. Our approach extends inverse RL to the vectored-valued setting which imposes a much weaker assumption than previous studies. The vectorized rewards incorporate Pareto optimality, which is a powerful tool to explain a wide range of behavior by its optimality. For practical assessment, our algorithm is tested on the World of Warcraft Avatar History dataset spanning three years of the gameplay. Our experiments demonstrate the improvement over the scalarization-based methods on real-world problem settings.
======== Summary ============
The authors consider a setup where there is a set of trajectories (s_t, a_t, r_t) where r_t is a *vector* of rewards. They assume that each agent is trying to maximize \sum_t \gamma^t (\phi . r_t) where \phi is a preference vector that lives on the simplex. Their goal is to calculate \phi (and maybe also an optimal policy under \phi?). The 
The authors first prove that this problem can be decomposed into finding Q functions for optimal policies for each component of r_t individually, and then solving for \phi that rationalizes the trajectory of actions in terms of these Q functions. Given the entire collection of trajectories, they perform off-policy Q-learning on each component of r_t in order to learn the Q function for that component, and then use linear programming to solve for \phi based on these Q function.
========== Comments =============
I think it's a worthwhile direction to combine IRL with modeling a diversity of preferences among agents. I can imagine several reasons you might want to do this, but the authors are not clear what their goal is besides "to propose methods that can help to understand the intricacy and complexity of human motivations and their behaviors". Is the goal to do better policy prediction? To do better policy prediction conditional on \phi? To infer \phi to understand people's preferences from a social science perspective? These all seems reasonable but not sufficiently teased out in the work. (For comparison, IRL is typically - although not always - interested in learning the reward function in order to construct robust policies that maximize it). The authors also don't seem to solve a particular task of importance on the WoW dataset.
The theoretical approach seems sound, and I liked the way their algorithm was motivated and the way the problem was decomposed into off-policy Q-learning and then solving for \phi.
However, I found myself quite confused in the experimental section (4.3). The authors evaluate their approach by action prediction. Given the trajectories, is \phi computed for each player and then compute actions based on that value of \phi? Is \phi computed on the same trajectory data used for evaluation or a different subset? Or is action prediction performed in aggregate across the entire population? The experimental setup was never clarified for this (main) experiment.
I was also confused about the motivation for Figure 2 and Appendix D. The authors are showing that their predictions about which reward is motivating the players is consistent with external factors. But wouldn't you see the same thing if you just plotted the observed *rewards* themselves? E.g. players in a guild will achieve more Relationship reward. 
The proposed approach takes the vector of reward, learns which actions are consistent with achieving each reward, then infers from the actions which reward is trying to be achieved. What advantages does this have vs. just looking at the empirical trajectory of rewards for each player/group?
I can certainly imagine that the IRL approach has certain advantages over looking at the empirical reward stream, but the authors have not talked about this nor compared against it experimentally.
The writing could also use some improvement for a future iteration, I've listed a few points below:
pg.1, Neither Brown & Sandholm nor Moravcik et al use "RL algorithms"
pg.1, Finn et al unmatched )
pg.1, "a scalar reward despite observed or not" -> "a scalar reward whether observed or not"
pg.2, "Either the range of" -> "Both the range of" (and this sentence needs further cleanup)
pg.2, "which records the pathing of players" ??
Theorem 3: "each of the set e_i has an unique element..." This isn't clear. I think you mean "For each e_i there is a unique vector v^\pi(s) for all \pi \in \Pi_{e_i} . The equality holds if these vectors are distinct for each e_i".
pg. 5 "If otherwise all elements in \phi are generative" how can they be negative if they are on the simplex?
pg.5 "we do not perform any scalarization on the reward...the model assumption is easier to be satisfied" I think this is a strange comparison to IRL because in IRL you're trying to find a (possibly parametric) function (s,a) -> R, whereas here you're *given* the vector R and are trying to find \phi. So while you have more degrees of freedom by adding \phi, you lose the original degrees of freedom in the reward function.