Showing versus Doing: Teaching by Demonstration
People often learn from others' demonstrations, and inverse reinforcement learning (IRL) techniques have realized this capacity in machines. In contrast, teaching by demonstration has been less well studied computationally. Here, we develop a Bayesian model for teaching by demonstration. Stark differences arise when demonstrators are intentionally teaching (i.e. showing) a task versus simply performing (i.e. doing) a task. In two experiments, we show that human participants modify their teaching behavior consistent with the predictions of our model. Further, we show that even standard IRL algorithms benefit when learning from showing versus doing.
This paper addresses the subject of teaching by demonstration. In particular, it asks the following question: When teaching from demonstration, is there a better way to form trajectories than following the optimal policy?  In addressing this question, the authors combine two earlier strands of research in the literature, inverse reinforcement learning and Bayesian pedagogy, to introduce a new model called pedagogical inverse reinforcement learning.  They perform two experiments with human subjects on two simple, deterministic gridworld problems, using a tabular and a feature-based reward function. The authors observe that people differ in the trajectories they produce when performing the task for themselves (to maximize return) and when demonstrating the task to others (so that they can maximize return as well). 
The subject of this paper (teaching by demonstration) is important and relevant for NIPS.   The primary questions that the authors ask are the following:  -- How might showing how to do a task differ from merely doing it?   -- How can inverse RL algorithms benefit from intentional teaching?   These questions are novel and this line of research can potentially have high impact.  The model introduced by the authors is a straightforward combination of two pieces of earlier work on inverse reinforcement learning and on Bayesian pedagogy. From this model, the authors make certain predictions, then conduct two experiments with human subjects to evaluate whether these predictions are observed in behavioral data.   In the paragraph that starts on line 124, the authors state that the pedagogical model makes two types of qualitative predictions. The first prediction is that certain trajectories show "clearer" paths than others. The second prediction is that certain "looping" behaviors will be observed. I have a number of concerns regarding these qualitative predictions.   My first concern is that these predictions are stated only verbally, not mathematically, and it is not entirely clear what the authors mean by them. For example, given two paths to the goal, how can we determine which one is a "clearer" path? Does the answer generalize to stochastic domains? Second, I do not follow how these predictions follow from the model in section 2.3. It would be useful if the authors can show how their model leads to these predictions (for example, can they derive them analytically?). It would also improve the paper to state the predictions clearly with no ambiguity. Third, it is not clear how the two types of predicted behaviors are actually useful for learning from demonstrations. For example, given that the goal states are distinct and far from each other, why is it desirable that the paths do not intersect?   The analysis of the two experiments are not as rigorous as one would like, and some experimental details are left out, which makes it difficult to interpret the results. Detailed comments below.   EXPERIMENT 1  1. What was the role of the learning block? From the text, I understand that in both the "Do" and "Show" conditions, the subjects were explicitly shown where the reward is. And if there were any participants learning from demonstration (the text is not clear on this), these participants observed deterministic trajectories generated by people who knew where the reward is, which means that they could observe the reward location indirectly, without having to learn.  2. Was the length of the trajectory important? That is, were shorter trajectories to the goal state more rewarding than longer trajectories? How was that information conveyed to the subjects? More generally, what exactly was the underlying reward function?  3. On line 148, the authors write that the participants in the "show" condition were more likely to choose paths that helped disambiguate their goal as compared to those in the "do" condition. As noted above, it is not clear why this type of disambiguation is useful. Was it indeed useful to the participants who learned by demonstration (if they did exist; see comment 1 above)? That is, was there a difference in the success rates of the participants who learned from demonstration when presented with the different trajectories?  4. What does M denote on line 151 (also used later)? I assume it is mean of something, but of what?  5. I do not follow the sentence that starts on line 152: "For trials where..." How does the model predict marginal significance?  6. Based on lines 145-146, I conclude that each participant completed 9 trials and that these nine trials used the 3 tasks shown in Figure 1, repeating each task 3 times. Is this correct? If yes, what is the rest of the data (Figure 1 shows only 3 trials)? If not, the paper should describe all 9 trials.  7. Why does the model predict loping behavior when % is the goal? (as stated in Figure 1 caption)  8. I do not see an analysis of the % goal. The difference  between the "do" and "show" conditions in this condition is much smaller than in the & condition. Why is that? Could there be an order effect?  9. Figure 1 left-hand column is showing trajectories from only 2 of the standard planning algorithms. This is misleading because these two trajectories are not special in any way. Given that the domain is deterministic, a number of other trajectories exist that collect exactly the same return as the two trajectories that are shown. On a related note, on the second column from the left, only the two model trajectories with the highest probabilities are presented --- it would be much more informative to see a larger number of trajectories, along with the actual probabilities assigned to them by the model. This information would show, for instance, whether there is a third trajectory with identical probability or whether there is a large gap in probability between the first and the second trajectory.  10. In section 3.3, it would be useful to see a high-level discussion of the overlap between model predictions and human behavior.   EXPERIMENT 2  11. There is an inconsistency in the terminal reward given. Is it 5 (line 196) or 10 (line 178)?  12. It would be nice to see a more direct link from the pedagogical model to the specific types of trajectories described in lines 187-190. (I have made similar comments earlier regarding Experiment 1.)  13. Columns 2 and 3 in Figure 2: See comment 9 above.   14. Line 211: "potentially suboptimal paths": Longer paths on safe tiles are *not* suboptimal; they collect exactly the same return as the shorter paths in safe tiles (unless it was communicated to the participants in some way that shorter trajectories are better, but I see no indication of this)  15. Assertions in lines 212-214 should be backed up with a quantitative analysis.  16. Line 217: The choice of alpha=1.5 should be discussed. In addition, it would be useful to have an analysis of the sensitivity of the results to the value of this parameter.  17. Lines 217-221: This introduces a difficulty in interpreting the results and should be discussed in some detail. In addition, using gamma=1 but adding a small negative reward for each action would place the two approaches at equal footing, is this not correct?    18. Most numbers in Table 1 are very small (except for the 0A condition), indicating that the observed trajectories are not a good fit to either model. Furthermore, the overall difference between "show" and "do" conditions is very small (as deduced from the last column in Table 1: about 0.02). Consequently, I do not think that the statements in lines 222-228 are a fair and accurate description of the results in Table 1.  19. Lines 237-239: Text is inconsistent with the numbers shown in parenthesis.   OTHER  -- Mathematical symbols in Equation 2 should be defined.  -- Algorithm 1, line 5: set pi (initialized on line 1) is still empty so this line would lead to j being assigned to the empty set.