Teaching Inverse Reinforcement Learners via Features and Demonstrations
Learning near-optimal behaviour from an expert's demonstrations typically relies on the assumption that the learner knows the features that the true reward function depends on. In this paper, we study the problem of learning from demonstrations in the setting where this is not the case, i.e., where there is a mismatch between the worldviews of the learner and the expert. We introduce a natural quantity, the teaching risk, which measures the potential suboptimality of policies that look optimal to the learner in this setting. We show that bounds on the teaching risk guarantee that the learner is able to find a near-optimal policy using standard algorithms based on inverse reinforcement learning. Based on these findings, we suggest a teaching scheme in which the expert can decrease the teaching risk by updating the learner's worldview, and thus ultimately enable her to find a near-optimal policy.
After response:  The authors' response helped clarify the computability of the risk metric for me. More experiments and discussion would improve the paper but after reading the response and other reviews I am raising my score.  *****************************  This paper introduces the problem of feature mismatch between a learner and teacher in the inverse RL problem (IRL). The problem is that if the learner and teacher represent the reward with different features than the learner may not be able to learn the teacher's reward function and thus perform suboptimally if it attempts to learn by matching feature counts. The paper quantifies this problem by defining the "teaching risk" and demonstrates that minimizing this quantity can improve learner performance. An algorithm is introduced that uses this quantity to select new features for the learner to use and it is shown to empirically lead to the learner being able to learn with less features than other baselines.  This paper presents an interesting problem that is likely to happen in real world applications of IRL. Furthermore, quantifying this problem in a way that corresponds to performance loss is interesting and is shown to be useful (though I have some questions about computability). To the best of my knowledge, this paper is the first to study this problem in the IRL setting and opens up an interesting direction for future research. My main concern with the paper is that it lacks clarity in some places (see below) and has little discussion of results or future work. Overall, I lean towards not accepting this paper but think it could be a stronger submission with appropriate revision.  My main technical concern is the computability of the teaching risk. Is the teaching risk computable in practice or is it just theoretically interesting? IRL is usually motivated by situations where w* is unknown but computing \rho seems to require w*. If w* is unknown, how can we compute the teaching risk in practice?  In terms of clarity, the paper relies on figures to make several of its key points. However, the description of the figures is only contained in the main text. The figures should have a complete description of their content (preferably in the caption). Right now the description is interspersed with remarks on what they demonstrate. It should be clear what is being shown before the reader is asked to accept conclusions about what they show.  In my opinion, figures 1, 2, and 3 don't illustrate much. It would be helpful for Figure 3 if the figure was described instead of simply saying, "Figure 3 shows why proposition 1 is true."  Where possible, it would help to see intuition before equations. For example, in "Basic Teaching Strategy" (lines 123-128) it would be useful to have the description of (1) before (1).   The paper lacks any discussion of future work and has only limited discussion of the empirical results. Since the paper is proposing a new problem, it would be good to have the paper also outline what the next steps are in addressing the problem.  Minor comments: 18: an reinforcement -> a reinforcement 62: recently been recently 67: have -> has if using citation number instead of names. Citations: Using numbers in place of names means the reader has to constantly turn to the references section. Saying "Name et al." is easier on the reader. 172: is vanishes -> vanishes 179 (and elsewhere): garantee -> guarantee 187 (and elsewhere): the use of "resp." is sometimes hard to parse what is being referred to. 283: better a -> better than a 273: optiaml -> optimal Supplementary material: It would be useful to have some section headers bolded. Line 140: policies pi_0, pi_1 have not been introduced at this point. Line 148: Is something missing in the definition of ker A^L