Scalable Lévy Process Priors for Spectral Kernel Learning
Gaussian processes are rich distributions over functions, with generalization properties determined by a kernel function. When used for long-range extrapolation, predictions are particularly sensitive to the choice of kernel parameters. It is therefore critical to account for kernel uncertainty in our predictive distributions. We propose a distribution over kernels formed by modelling a spectral mixture density with a Lévy process. The resulting distribution has support for all stationary covariances-including the popular RBF, periodic, and Matérn kernelscombined with inductive biases which enable automatic and data efficient learning, long-range extrapolation, and state of the art predictive performance. The proposed model also presents an approach to spectral regularization, as the Lévy process introduces a sparsity-inducing prior over mixture components, allowing automatic selection over model order and pruning of extraneous components. We exploit the algebraic structure of the proposed process for O(n) training and O(1) predictions. We perform extrapolations having reasonable uncertainty estimates on several benchmarks, show that the proposed model can recover flexible ground truth covariances and that it is robust to errors in initialization.
This paper models the spectrum of a Gaussian process kernel with a location-scale mixture of Lalpacians, using the LARX method of [1] for a non-parametric prior over the coefficients. It is claimed that the levy prior promotes sparse solutions. A reverse jump mcmc method is proposed, following [1]. Inference is accelerated using a SKI approximation from [2], and results are presented on the Airline dataset and also a dataset from natural sounds.
The paper is nicely written and the idea is interesting and novel, but the evaluation of the method is extremely weak. In particular, there are no quantitive comparisons, nor any mention of other kernel learning approaches (e.g. compositional kernels [3]) or other spectral approaches to GP regression (e.g. sparse spectrum GPs [5])
The first experiment presented is actually not the approach of the paper, but instead the method of [1]. It is unclear how this provides evidence for the claims of the novel contributions of the paper. The second experiment shows (qualitatively) that the model can fit a sample from its prior better than a different model. It is hardly surprising that the method of [5] fares less well on this task as Gaussian model over the spectrum is misspecified. The third experiment provides a comparison with the method of [5], but from the graph it is extremely unclear which of the two method is better, and no results on held-out data are presented. The final experiment serves only as a demonstration of scalability as the original data from the in-filled section is not shown, and the only evaluation given is that ‘the reconstucted signal has no discernable artifacts when played back as sound’. Presumably the sound file will be included in the final version, but even so, it is not easy to assess the significance of such evidence. Do other more simple approaches achieve this property? (Incidentally, the quoted sentence appears not to have been spell-checked - fortunately it seems to be the only one!)
Beyond the experiments, I feel the paper should be improved by distinguishing its key contributions. It appears that there are two: the use of laplacian basis functions to model the fourier spectrum (as opposed to Gaussian in [5]), and the use of the levy prior. Is maximum likelihood over the laplacian features as effective as [5]? It should be made clear whether this is a modelling implication or chosen for convenience. Secondly, it should be explained in what circumstances a ML approach leads to overfitting, requiring the use of the levy prior. 
Finally, I feel that the use of ‘scalable’ in the title is a little misleading. The kernel interpolation approach is an approximation (the quality of which is not discussed), and does not extend readily to high dimensions.
In summary, to improve this paper I suggest the results section is made more quantitive and emphasis on scalability is dropped and replaced by a focus on exactly why such priors are useful over a simpler ML approach.
[1] Wolpert, R.L., Clyde, M.A., and Tu, C. Stochastic expansions using continuous dictionaries: Levy adaptive regression kernels. The Annals of Statistics, 39(4):1916–1962, 2011.
[2] Wilson, Andrew Gordon and Nickisch, Hannes. Kernel interpolation for scalable structured Gaussian processes (KISS-GP). International Conference on Machine Learning (ICML), 2015.
[3] Duvenaud, David, et al. "Structure discovery in nonparametric regression through compositional kernel search." arXiv preprint arXiv:1302.4922 (2013).
[4] Quinonero-Candela, Joaquin, Carl Edward Rasmussen, and Annbal R. Figueiras-Vidal. "Sparse spectrum Gaussian process regression." Journal of Machine Learning Research 11.Jun (2010): 1865-1881.
[5] Wilson, Andrew, and Ryan Adams. "Gaussian process kernels for pattern discovery and extrapolation." Proceedings of the 30th International Conference on Machine Learning (ICML-13). 2013.
APA