Variational Inference for Mahalanobis Distance Metrics in Gaussian Process Regression
We introduce a novel variational method that allows to approximately integrate out kernel hyperparameters, such as length-scales, in Gaussian process regression. This approach consists of a novel variant of the variational framework that has been recently developed for the Gaussian process latent variable model which additionally makes use of a standardised representation of the Gaussian process. We consider this technique for learning Mahalanobis distance metrics in a Gaussian process regression setting and provide experimental evaluations and comparisons with existing methods by considering datasets with high-dimensional inputs.
The paper proposes a variational bound on the length 
      scale parameters of square-exponential-kernel Gaussian process regression 
      models. The main idea is to separate the function to be inferred into a 
      "standardised" sample from a unit-length-scale square-exponential kernel, 
      and a linear scaling map of that latent function, then to impose 
      factorisation between these two objects via a variational bound. 
      The paper is well written. It uses clear language and provides a 
      compact introduction to previous work. To my knowledge, the idea is novel. 
      Length scales are a perennial problem in kernel regression models, so the 
      promise of a lightweight, efficient and effective approximate 
      probabilistic solution (as opposed to the existing alternative of MCMC 
      sampling) is a significant contribution. My main concern is the 
      experimental evaluation. The experiments presented in Figure 2 do a good 
      job of arguing that the new method improves on SPGP-DR. What they don't 
      show is whether the new method is actually good at capturing posterior 
      mass in the length scales. The paper's point would be strengthened by a 
      comparison between the independent marginals (Eq. 11) inferred by the new 
      method and those found by a "gold standard", e.g. from an MCMC sampler. 
      The variational bound contains some strong factorization constraints (Eq. 
      11), and such are known to cause overconfidence. This may, or may not be a 
      problem here. I would be grateful if the authors could clarify the 
      following points in the feedback: * I understand, from line 293 
      that the "full GP" is type-II maximum likelihood, with the ARD kernel? I'm 
      surprised that this model is consistently better than the probabilistic, 
      Mahalonobis SPGP-DR on the puma dataset. Section 3.3 does not seem to 
      explain this. * lines 380 and lines 404-409 argue that the new 
      method is fast. But none of the plots quote runtimes. Of course, runtime 
      comparisons must be taken with a grain of salt, but they are still 
      interesting. Could you quote a few numbers? How fast is VDMGP compared to 
      the full GP and to SPGP-DR, for small and large datasets? Finally, 
      a minor point about the conclusion: You are suggesting, for future work, 
      to use locally varying length scales in the kernel. This is not part of 
      the paper so not a core point of my review, but as far as I understand, 
      the generalization you propose does not actually give a valid covariance 
      function. See Mark Gibbs' PhD thesis (Cambridge), page 18, and Section 
      3.10.3 (page 46 and following), which also gives ideas for alternate 
      approaches.