Optimal Teaching for Limited-Capacity Human Learners
Basic decisions, such as judging a person as a friend or foe, involve categorizing novel stimuli. Recent work finds that people's category judgments are guided by a small set of examples that are retrieved from memory at decision time. This limited and stochastic retrieval places limits on human performance for probabilistic classification decisions. In light of this capacity limitation, recent work finds that idealizing training items, such that the saliency of ambiguous cases is reduced, improves human performance on novel test items. One shortcoming of previous work in idealization is that category distributions were idealized in an ad hoc or heuristic fashion. In this contribution, we take a first principles approach to constructing idealized training sets. We apply a machine teaching procedure to a cognitive model that is either limited capacity (as humans are) or unlimited capacity (as most machine learning systems are). As predicted, we find that the machine teacher recommends idealized training sets. We also find that human learners perform best when training recommendations from the machine teacher are based on a limited-capacity model. As predicted, to the extent that the learning model used by the machine teacher conforms to the true nature of human learners, the recommendations of the machine teacher prove effective. Our results provide a normative basis (given capacity constraints) for idealization procedures and offer a novel selection procedure for models of human learning.
The work presented here is solid and innovative from a cognitive scienceperspective, and the manuscript is well written and for the most part clear.  Ihave some issues and questions for the authors, which I suspect can beaddressed.Major comments:On line 031 of the abstract, the authos conclude, "We find that the optimalteacher recommends idealized training sets."  I believe that the idealizationof training sets is a presumption of the 'optimal' teacher, so the teacher isreally an 'optimal idealized-training-set' teacher.  I would be happy if theauthors simply stated at the outset that they were going to limit theirexploration to a search for idealized training sets, based on pastpsychological research.  Because they seem unwilling to make such a statement,they introduce idealization in a slippery backdoor kind of way (see line 175,the sentence beginning "We do not have evidence...")It is true that the various idealized training sets outperform one particularnonidealized set (the random set), but I don't think the right comparisonshave been done to conclude that idealization is the optimal strategy forlimited-capacity human learners.It would be useful to say a few words about what relevance idealization has totraining real-world tasks.  Idealization is a different idea than simply usingprototypes: idealization involves labeling nondeterministic stimuli with theirpredominant classification.  But with naturalistic tasks, one may not be ableto ascertain the degree of ambiguity of a stimulus.  Although the paper is notabout idealization per se, it presumes that the teacher can idealize stimuli.The authors report that the optimal teacher selects clump-far examplesfor a limited-capacity learner (line 222). I would like to know more aboutthese clump-far sets.  Figure 2A shows that the 'far' clump is part way but notall the way to the stimulus dimension extrema.  Why not stimuli furtherfrom the boundary?  What's the trade off that lands the 'far' stimuli at around.15 and .85?  Is the optimal set really 5 identical examples adjacent to5 identical examples, or is that simply how the figure is drawn?I didn't go back and look at the Nosofsky & Palmieri Psych Rev paper (reference[17]), but it's difficult to intuit the claim that \gamma relates to memorycapacity (line 110).  My intuitions are further shot by the comment (lines227-299): "The high-capacity GCM...is sensitive only to the placement oftraining items adjacent to the decision boundary".  This doesn't sound like ahigh capacity model.Until the last paragraph of the article (line 429), the authors did nottouch on an issue that bothered me from the outset: that ordereffects are ignored.  To the degree that human memory is capacity limited,it is also recency biased, and the fact that recency doesn't come into playfurther weakens my acceptance of the low-lambda GCM as a model oflimited-capacity instance-based learning.MINOR COMMENTS:line 025: In the abstract and the introduction, the authors comment thatidealization runs counter to ML practice, where one aims to match statistics ofthe training and test sets.  However, with limited training set size, my hunchis that idealization would benefit ML classifiers as well on the particulartask studied in this paper.  For example, comparing a 'spread' idealizedtraining set to a 'spread'  non-idealized set, It seems likely that thevariability in the inferred decision boundary will be smaller with theidealized set.057: What does "idealized in an ad hoc or heuristic fashion" mean?192, "discuss the experiment with human learners": There's ambiguity in theprepositional phrase attachment. You might want to re-word.205: I'm lost on the notation. Where are the empirical probabilities shown?Is the $y_j = 1$ term in the equation for $f^{(cond)}$ the observed outcomeof an experimental trial?Figure 4: It might be a good idea to zoom in to focus on the 50-75%performance range.