Efficient High Dimensional Bayesian Optimization with Additivity and Quadrature Fourier Features
We develop an efficient and provably no-regret Bayesian optimization (BO) algorithm for optimization of black-box functions in high dimensions. We assume a generalized additive model with possibly overlapping variable groups. When the groups do not overlap, we are able to provide the first provably no-regret polynomial time (in the number of evaluations of the acquisition function) algorithm for solving high dimensional BO. To make the optimization efficient and feasible, we introduce a novel deterministic Fourier Features approximation based on numerical integration with detailed analysis for the squared exponential kernel. The error of this approximation decreases exponentially with the number of features, and allows for a precise approximation of both posterior mean and variance. In addition, the kernel matrix inversion improves in its complexity from cubic to essentially linear in the number of data points measured in basic arithmetic operations.
In this paper, the authors introduce an improvement to the standard random fourier features (RFF) scheme for kernel approximation. Rather than rely on Monte Carlo estimates of the integral involved in Bochner's theorem, the authors derive features from a Gauss-Hermite quadrature estimate of the integral. The result is a kernel estimation scheme that convergences uniformly over the domain quickly. The authors demonstrate theoretically that the improved uniform convergence results in theoretical guarantees on the convergence of both the posterior moments of the Gaussian process and the cumulative regret of Bayesian optimization.  Overall, I quite like the paper. It tackles an issue that really does come up in practice: I have often found random Fourier features to produce unsatisfactory sample paths from the posterior Gaussian process. A method that solves this issue both in practice and provides theoretical convergence guarantees is quite nice. In my opinion, this is good work. I do have a few questions and comments for the authors that I detail below.  Right now, my biggest complaint is that the paper largely ignores the existence of approximate GP methods that aren't based on finite basis expansions. Inducing point methods (e.g. Titsias et al., 2009) can lead to accelerated sampling procedures. When performing stochastic variational inference (Hensman et al., 2013, Wilson et al., 2016), the time complexity required to perform sampling can be as low as O(tm^2) depending on the kernel approximation used. Pleiss et al., 2018 introduces a procedure that achieves O(t) time approximate sampling and applies it to the additive BayesOpt setting. As far as I am aware, theory does not exist demonstrating a cumulative regret bound for Bayesian optimization using sparse GP methods, but theory does exist demonstrating convergence of the posterior moments.   I assume QFF is compatible with other acquisition functions that can be computed using sample paths from the posterior Gaussian process (e.g., max value entropy search (Wang et al., 2017), predictive entropy search (Hernández-Lobato et al., 2014), knowledge gradient (Scott et al., 2009), ...). Have you run experiments with these? The max value entropy search paper in particular discusses another (albeit fairly coarse approximate) approach to sampling directly from the maximum value distribution, p(y*|D).  Finally, can the authors comment a bit on their focus on the high dimensional Bayesian optimization setting? In my mind, the primary contribution of the paper is the provably good kernel approximation that solves a number of issues with the RFF approach. Is this story largely due to a decreased applicability of QFF to high dimensional sampling? Is the regret bound in Theorem 3 substantively different as a result of using QFF? It appears to have the same flavor as has been demonstrated before for additive Bayesian optimization, where cumulative regret depends linearly on the number of additive components and exponentially on the size of the largest additive component. If QFF is limited to usage with additive structure GPs, it's worth noting that this setting also appears in deep kernel learning (DKL, Wilson et al., 2016) where it is standard practice to assume additive structure at the end of the neural network. QFF could therefore be used as a drop-in replacement for kernel approximation schemes in DKL.