Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior
Inferring intent from observed behavior has been studied extensively within the frameworks of Bayesian inverse planning and inverse reinforcement learning. These methods infer a goal or reward function that best explains the actions of the observed agent, typically a human demonstrator. Another agent can use this inferred intent to predict, imitate, or assist the human user. However, a central assumption in inverse reinforcement learning is that the demonstrator is close to optimal. While models of suboptimal behavior exist, they typically assume that suboptimal actions are the result of some type of random noise or a known cognitive bias, like temporal inconsistency. In this paper, we take an alternative approach, and model suboptimal behavior as the result of internal model misspecification: the reason that user actions might deviate from near-optimal actions is that the user has an incorrect set of beliefs about the rules -the dynamics -governing how actions affect the environment. Our insight is that while demonstrated actions may be suboptimal in the real world, they may actually be near-optimal with respect to the user's internal model of the dynamics. By estimating these internal beliefs from observed behavior, we arrive at a new method for inferring intent. We demonstrate in simulation and in a user study with 12 participants that this approach enables us to more accurately model human intent, and can be used in a variety of applications, including offering assistance in a shared autonomy framework and inferring human preferences.
The paper investigates the problem of inferring an agent's belief of the system dynamics of an MDP, given demonstrations of its behavior and the reward function it was optimizing. Knowledge of this internal belief can be used for Inverse Reinforcement Learning of an unknown task in the same environment. Furthermore, given the action provided by the agent, its intended action on the true dynamics can be inferred. This allows for assistive tele-operation, by applying the intended actions to the system instead of the provided ones. The proposed method models the agent using the model derived in maximum causal entropy inverse reinforcement learning. The internal belief of the system dynamics is inferred by maximizing the likelihood of the demonstrations with respect to the parameters of the dynamic model. Human studies on the lunar lander Atari game demonstrate that such assistive systems based on inferred internal models can help the user to solve the given task.  Related problems in an MDP are RL, which learns behavior from dynamics and reward, as well as IRL, which learns reward from dynamics and behavior. This work tackles the problem of learning dynamics from behavior and reward which has gotten little attention from the community. There has been some prior work on this problem that was also mentioned in the paper. Herman et al. (2016) considered the problem of simultaneously inferring dynamics and reward function from expert demonstrations while also building on MaxCausalEnt-IRL. Although, the work by Herman et al. allows for internal models that differ from the true dynamics, they focused on inferring the true dynamics and assumed in the experiments that the internal model was correct. Golub et al. (2013) infer an internal model from reward function and dynamics. They derived an EM-algorithm that they applied for learning the internal model of Rhesus monkeys that were using a brain computer interface. However they consider a slightly different scenario where the subject's internal estimate of the state differs from the true state due to sensory feedback delay. The paper under review focuses on the internal model (and assumes that the true dynamics are known for the action intent prior) and in contrast to previous work puts more focus on the use cases. Using an internal model to correct the user's actions is novel to my knowledge and could be useful for real applications like tele-operating a robot. Also the application to IRL is interesting, however I think this scenario is less relevant, because we assume that 1) we want to infer the reward function of an agent that 2) is willing to perform demonstrations on different tasks and we 3) assume that its internal model is significantly different from the true dynamics.   The proposed method is based on a sound optimization problem by maximizing the likelihood of the demonstrations assuming the expert model of MaxCausalEnt-IRL. Yet, I think it would be more principled to derive both the expert model as well as the objective from a single optimization problem (e.g. similar to MaxEnt Reinforcement Learning but optimizing dynamics rather than policy).  This might also avoid the need for the penalty method, which is a simple but rather inefficient and unstable approach. The proposed method does in general tend to prefer the easy route, e.g. by assuming deterministic dynamics (although humans seem to take uncertainty into account [1]) and by using uniform sampling in section 3.1. which should lead to rather high variance. Nevertheless, the method is still sound and, in my opinion, the method itself is not the main contribution of the paper anyway. Indeed, it is more important to raise the awareness and show the usefulness of such little explored research branches, which is I think the main contribution of the proposed paper.  The paper is well written and easy to follow. However, I think it would be better if the experiments would be better separated, e.g. by removing everything relating to the lunar lander game from section 5.1 and calling the section something like "Gridworld experiments". The gridworld experiments should be briefly described also in the paper itself.  Questions to the authors: For learning the internal model, we need the user to optimize a given reward function. For some problems, specifying a reward is easy (e.g. Atari games). However, how would you make a human optimize a reward according to maxent-model for a real application?  Concerning the action intend prior: Wouldn't it be simpler to 1) solve the MDP using the true dynamics and 2) compute the mapping (f_theta) from actions to intended actions as the maximum likelihood conditional distribution?  How are the demonstrations on the gridworld collected? If whole trajectories were collected at once, I assume that the user knew how the dynamics are scrambled since the proposed method can not handle changing beliefs. However, if the user knew the true dynamics, why would his internal belief differ? Wouldn't the sub-optimality of the user's behavior be caused solely by its inability to choose the correct actions under real-time constraints? Or were the trajectories collected by independently querying the users action for a given state without revealing the consequences of this action?  [1] Belousov, Boris, et al. "Catching heuristics are optimal control policies." Advances in Neural Information Processing Systems. 2016.  Edit: I read the the other rebuttal / reviews and did not see a reason to change my assessment.