Machine Teaching for Bayesian Learners in the Exponential Family
What if there is a teacher who knows the learning goal and wants to design good training data for a machine learner? We propose an optimal teaching framework aimed at learners who employ Bayesian models. Our framework is expressed as an optimization problem over teaching examples that balance the future loss of the learner and the effort of the teacher. This optimization problem is in general hard. In the case where the learner employs conjugate exponential family models, we present an approximate algorithm for finding the optimal teaching set. Our algorithm optimizes the aggregate sufficient statistics, then unpacks them into actual teaching examples. We give several examples to illustrate our framework.
The authors generalize the teaching dimension theory 
      [Goldman & Kearns, 95] to the family of Bayesian Learners (and 
      hypothetically to noisy teachers) -- although they restrict any actual 
      analysis to exponential families of distributions (or at least 
      distributions with conjugate priors -- which admittedly much of machine 
      learning does). The basic model minimizes the combination of loss from the 
      resulting learner hypothesis and effort of the teacher. Based of the basic 
      idea of selecting a dataset that has the correct sufficient statistics, 
      they first provide a solution to the 1D threshold function (under two 
      different scenarios), a restricted model selection problem (one of two 
      Gaussian distributions), and eventually generalize this to teaching 
      Bayesian Learners of the family of Exponential Distributions. 
      Overall, the paper is very clear, although quite a bit of space is 
      spent on trivial examples while the exponential family section is by far 
      the most interesting and practical. My biggest concern with this paper is 
      that while it does improve the teaching dimension literature, it seems to 
      think it is making a bigger jump than it is (and sort of overselling its 
      claims a bit to be honest) as it is missing some reference to 
      statistics/cognitive science literature that have very similar results (I 
      am not even sure if “mostly studied from th query learning angle” is 
      entirely correct -- although it certainly has been studied from this 
      angle). For example, [Tenenbaum & Griffiths, 2001; Shafto & 
      Goodman, 2008; Rafferty & Griffiths, 2010]. There are clearly some 
      very strong connections here and the work should at least be acknowledged 
      (and I actually think could save some space for discussion on the 
      exponential families model) -- although I do admit that I like the 
      presentation better in this paper (but this is likely since I am not a 
      cognitive scientist). While I am not a cognitive scientist, I am 
      also somewhat concerned about the claims regarding the application to 
      human learning. Given the authors knowledge of query learning, I am 
      actually somewhat surprised by this claim. My understanding is that the 
      current theory is that humans are not passive Bayesian learners -- not 
      only is the presentation of the examples by the teacher not iid, but it is 
      influenced by feedback from the students (which I suppose could 
      hypothetically be modeled as some sampling from the distribution of the 
      student) and if nothing else, the prior of the student are certainly not 
      known (although, again, I suppose they could be probed). That 
      being said, I think it is a nice work overall and would be interesting to 
      the NIPS community (in particular relative to the machine learning 
      community in general). It is a very clean presentation, made me think a 
      bit, and would likely influence my work -- which means it would likely be 
      of interest to the NIPS community (even if I think it is a bit “in 
      progress” with respect to contextualizing within related work).