Frank-Wolfe Bayesian Quadrature: Probabilistic Integration with Theoretical Guarantees
There is renewed interest in formulating integration as a statistical inference problem, motivated by obtaining a full distribution over numerical error that can be propagated through subsequent computation. Current methods, such as Bayesian Quadrature, demonstrate impressive empirical performance but lack theoretical analysis. An important challenge is therefore to reconcile these probabilistic integrators with rigorous convergence guarantees. In this paper, we present the first probabilistic integrator that admits such theoretical treatment, called Frank-Wolfe Bayesian Quadrature (FWBQ). Under FWBQ, convergence to the true value of the integral is shown to be up to exponential and posterior contraction rates are proven to be up to super-exponential. In simulations, FWBQ is competitive with state-of-the-art methods and out-performs alternatives based on Frank-Wolfe optimisation. Our approach is applied to successfully quantify numerical error in the solution to a challenging Bayesian model choice problem in cellular biology.
Update after response: with the clarifications mentioned, I'm very happy with the state of this paper as a theoretical contribution to an important problem, and am increasing my score from 7 to 8. ~~~ This paper unifies the Frank-Wolfe based approach for numerical integration with Bayesian Quadrature, developing a numerical integration scheme that provides a full posterior distribution over the result as well as giving good convergence rates. This then also tells us that BQ with optimally-selected points has at least the same convergence rates.It seems that Theorem 1 says only that the BQ weights are no worse than FW weights, and then gives the same convergence rates as [2]. Thus this paper does not establish theoretically that FW(LS)BQ converges faster than FW(LS), but rather that it is no slower. This was not clear to me on first reading and perhaps should be noted more explicitly.The assumption of a finite-dimensional Hilbert space may be standard, but it makes the results somewhat hard to interpret. This assumption seems to come in only through [2, Prop. 1]; [2, Prop. 2] establishes that this same convergence in fact does *not* hold for an infinite-dimensional RKHS. Though section 4.1 notes that any kernel is in practice finite-dimensional, it seems reasonable from the proof of [2, Prop. 1] that as the dimension of the kernel becomes large, the constants C will become quite small and convergence will become slow. It would thus be informative to include (perhaps in the appendix) experiments with a truly finite-dimensional kernel to see if the line search methods perform better in that case. One interesting such experiment might be to use approximations to the exponentiated-quadratic kernel (e.g. random Fourier features) of increasing dimension.On line 279-280: This seems to imply that the main way kernel choice influences the convergence rate is through the constant R. But R = \sup_{x, y \in X} ||\Phi(x) - \Phi(y)||   = \sup_{x, y \in X} \sqrt{k(x, x) + k(y, y) - 2 k(x, y)}. Consider the exponentiated-quadratic kernel of section 4.1. When X is unbounded, R = \sqrt{2} \lambda. One can thus minimize R by minimizing \lambda, but of course that is not the best path to take, and indeed changing \lambda also changes the constants C. In general, some discussion of how C is affected by the kernel choice would be quite interesting.It seems that maintaining the constants ||f||, R, and so on in Theorems 1 and 2 would not be overly cumbersome; though C is of course difficult to determine, it at least satisfies a few simple scaling properties, and so this might provide added understanding in the behavior of the bound under different situations.Lines 345-346 note that "the weights for SBQ are optimal at each iteration, which is not the case for FWBQ and FWLSBQ." Given the design points, don't FW(LS)BQ and SBQ weight those points in the same way?It would be useful to briefly compare the computational costs of the various methods, both asymptotically (which is noted at various points through the paper but not fully compared) and in terms of actual runtimes.Relatedly, it seems that M(n) must grow exponentially in n for the stated FWLSBQ rate to hold, which seems like a substantial computational burden.The conjecture that the convergence rates also apply to SBQ seems reasonable. Given that, is there any practical situation in which FW(LS)BQ would be preferable to SBQ, or should it be used only by conservative practitioners seeking a guaranteed method?