Bayesian optimization for automated model selection
Despite the success of kernel-based nonparametric methods, kernel selection still requires considerable expertise, and is often described as a "black art." We present a sophisticated method for automatically searching for an appropriate kernel from an infinite space of potential choices. Previous efforts in this direction have focused on traversing a kernel grammar, only examining the data via computation of marginal likelihood. Our proposed search method is based on Bayesian optimization in model space, where we reason about model evidence as a function to be maximized. We explicitly reason about the data distribution and how it induces similarity between potential model choices in terms of the explanations they can offer for observed data. In this light, we construct a novel kernel between models to explain a given dataset. Our method is capable of finding a model that explains a given dataset well without any human assistance, often with fewer computations of model evidence than previous approaches, a claim we demonstrate empirically.
This paper introduces a novel approach to kernel choice, especially in the case of Gaussian processes, through Bayesian optimization. A very interesting kernel between kernels is proposed. While the paper focuses on GPs, the method could be used more generally, and the techniques developed (approximating the model evidence, a "kernel kernel") are potentially useful beyond this setting.
I think the clarity of your presentation can be strengthened.   1) Line 80 - hyperprameters -> hyperparameters 2) Line 66 - adopted -> adapter 3) Line 83 - "principle approach" -> principled approach 4) Footnote 1 - maybe I'm one of the authors who's to blame, but I'm perfectly comfortable thinking of model selection as hyperparameter learning. Sure, I'd like to integrate out the hyperparameters (and thus average over a class of models), but if I can't do that, then I'm happy to learn them and select a single model. Indeed, I'd question your claim that the Bayesian perspective is one of Bayesian model selection---you're just moving one more level up the hierarchy (hyper-hyper parameters!), but wouldn't it be even better if we could integrate over different models M? This might actually be possible within your framework---you're using Bayesian optimization to maximize a function, but the recent probabilistic numerics work on Bayesian quadrature suggests you could instead use BO to do integration! 5) I was a little confused about how you maximize the acquisition function---where does the set of candidate models C come from, and what exactly do you have to do for each candidate model. I think run an optimization to approximate the model evidence, is that right? If so, it's confusing me because the model evidence approximation isn't discussed until later -- page 5 while the acquisition function is discussed on page 3. 6) The priors you suggest are reasonable, though I wonder if it'd be better to use a distribution like Student t (see some work by Vehtari and colleagues) in place of a Normal. Do you have any citations to back up your choices? Why sd = 0.7? 7) I'd like a little more discussion in Section 4.3. Would you suggest using Laplace in place of the use of BIC in Duvenaud et al? You should provide a page number in Murphy, as I'm curious to know more about BIC vs Laplace. 8) Section 4.4 is very interesting. Eq. 6 looks like what happens when you create a cross-covariance of two SE kernels (e.g. Melkumyan and Ramos, IJCAI 2011) though of course the setup is different. I'm also curious whether if there's an alternative approach to creating a kernel kernel by considering the maximum mean discrepancy between the distributions. 9) After rereading Section 5.1, I'm still a little confused about what you're evaluating. You should more clearly say what your method is, and why/what the mean prediction / k-nearest neighbors regression are doing. I think what you've left implicit is exactly what the ground truth is to which you're comparing. Once you say that, and say what your method is doing, everything should become clear.