Cooperative Inverse Reinforcement Learning
For an autonomous system to be helpful to humans and to pose no unwarranted risks, it needs to align its values with those of the humans in its environment in such a way that its actions contribute to the maximization of value for the humans. We propose a formal definition of the value alignment problem as cooperative inverse reinforcement learning (CIRL). A CIRL problem is a cooperative, partialinformation game with two agents, human and robot; both are rewarded according to the human's reward function, but the robot does not initially know what this is. In contrast to classical IRL, where the human is assumed to act optimally in isolation, optimal CIRL solutions produce behaviors such as active teaching, active learning, and communicative actions that are more effective in achieving value alignment. We show that computing optimal joint policies in CIRL games can be reduced to solving a POMDP, prove that optimality in isolation is suboptimal in CIRL, and derive an approximate CIRL algorithm.
The paper presents a general framework for cooperative interaction between a human and a robot. This framework encompasses well-known settings such as inverse reinforcement learning (IRL) or optimal teaching. The authors argue that it can be cast as a POMDP problem and then present in more details how apprentice learning can be formalized in their framework. This leads to the following insight: if the human knows that the robot uses IRL, she may be better off not demonstrating with an optimal policy, but with a “best response” to the robot’s strategy. The authors also provide an approximation scheme in the case where rewards are expressed as linear combinations of state features and when the human demonstrate only one trajectory. Finally, experimental results in the navigation problem support the previous insight.       
           My opinion on this paper is quite mixed. On the one hand, the paper is generally clear. I like this general unifying framework and the insight it provides. On the other hand, the exposition could be made more precise and more rigorous. Some technical explanations and the proofs are a bit hand wavy. Besides, the theoretical results seem quite straightforward.                      Here are some more detailed comments:                      - Nash equilibrium is a concept developed in non-cooperative games. As the proposed framework is cooperative, the terminology from non-cooperative games may not be the best.                      - I find it strange to assume that both agents know the probability distribution over \theta. H doesn’t simply choose \theta? The model is a representation from the viewpoint of the robot and so maybe P_0 could simply be interpreted as its prior belief.                      - I think the authors should give the formal definition of the coordination POMDP to make things clearer. For instance, the explanations given in the paper and the supplementary material do not match: for instance, for the POMDP states (l.212, l.288 in paper and Def.2 in supplementary material).                      - In Sec.4.2, does the IRL approach also demonstrate only one trajectory? Assuming this is the case, could the experimental results be explained by the fact that only one trajectory is demonstrated?                      Some typos:           l.88: Should the histories be defined by S x (A^H x A^R x S)^*?           l.250: it gets           l.270: gives use the ability -> gives the ability?           l.272: although obvious, br is not defined           l.279: Remark 3 seems not to be really related to what precedes it.           l.287: a POMDP           l.321: Why is Figure placed on p.3?           l.326: The robot reward function -> The reward function           l.407: Kand -> K, and