The Human Kernel
Bayesian nonparametric models, such as Gaussian processes, provide a compelling framework for automatic statistical modelling: these models have a high degree of flexibility, and automatically calibrated complexity. However, automating human expertise remains elusive; for example, Gaussian processes with standard kernels struggle on function extrapolation problems that are trivial for human learners. In this paper, we create function extrapolation problems and acquire human responses, and then design a kernel learning framework to reverse engineer the inductive biases of human learners across a set of behavioral experiments. We use the learned kernels to gain psychological insights and to extrapolate in humanlike ways that go beyond traditional stationary and polynomial kernels. Finally, we investigate Occam's razor in human and Gaussian process based function learning.
The paper is fantastic in many manners and very well-written. I will focus on (hopefully) constructive criticism:There is too much packed into the paper. As such, there are some details that should be in the paper, but are not. For example, a reader should be able to replicate the modeling and human experiments based on reading the article alone, if she desired. I do not think I could do either based on only the article. Although the supplement helps, it does not help with the experiment presented in section 5. My main other criticism is that the psychological implications of the studies presented by the authors is sloppy at times. Does it really get at THE human kernel? Their own results seem to contradict this as different kernels were learned depending on the experiment (compare Figure 2o to Figure 3d). I also suspect the kernel will be highly domain-dependent. Also, the authors write "they are more confident in their predictions...". As far as I can tell, the researchers did not get individual confidence ratings and without that, it seems inappropriate to say that they are more confident. I would say that they is less variability between participants. Finally, I have one last concern with respect to the stimuli. As the authors note at the end, the nature of the stimuli typically used in the human function learning is very different than that used in the experiments. I'm not a fan of the standard human function learning stimuli, but you should mention that it is going to be different right away, as it is confusing to cognitive science readers that are familiar with the human function learning literature. Further, the author's stimulus presentation introduces a confound: they could be studying human inductive biases over 1-d curves in images rather than functions themselves. It is probably worth mentioning this limitation of this method.More minor concerns: Paragraph 2 of 4.1: I don't understand what the prediction kernel is. Also, why use RBFs if later you focus on RQs? Also also, you never define RQ... I found Figure 2 difficult to parse. It might be worth considering other ways of organizing the subplots. If you are interested in human rule learning in number sequences, you might consider citing and/or taking a look at Austerweil & Griffiths (2011). Seeking Confirmation is Rational for Deterministic Hypotheses. Cognitive Science. line 351-352: "Participant were" -> "Participants were" The sections for experiments are mislabeled in the supplementary materials (and also the last 2/3 experiments are missing).