Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
We propose a novel information-theoretic approach for Bayesian optimization called Predictive Entropy Search (PES). At each iteration, PES selects the next evaluation point that maximizes the expected information gained with respect to the global maximum. PES codifies this intractable acquisition function in terms of the expected reduction in the differential entropy of the predictive distribution. This reformulation allows PES to obtain approximations that are both more accurate and efficient than other alternatives such as Entropy Search (ES). Furthermore, PES can easily perform a fully Bayesian treatment of the model hyperparameters while ES cannot. We evaluate PES in both synthetic and realworld applications, including optimization problems in machine learning, finance, biotechnology, and robotics. We show that the increased accuracy of PES leads to significant gains in optimization performance.
This manuscript proposes a new acquisition function for use in Bayesian optimization with a Gaussian process belief on a latent objective function.  The proposed criterion is a rewriting of the previously suggested entropy search (ES) objective [Hennig and Schuler, JMLR 2012], which seeks to maximize the posterior entropy of the _location_ of the global maximum x*, H[x* | D].  We sequentially greedily select the next point x' in the domain of the objective which maximizes the expected posterior entropy H[x* | D'], where D' is the current data augmented by the proposed new observation (x', y').  This requires marginalizing the unknown observation value y' and, if we are to be fully Bayesian, the hyperparameters of the model, \theta.  The authors first write out the mutual information between x* and y' (given D) in the way that it would most often be encountered in an active learning context (and how entropy search was originally posed by Hennig and Schuler).  The expression is unwieldy due to the marginalization of y' and the repeated need to approximate H[x* | D'], which requires many approximations.The authors instead propose to use a trick that has been previously used in other similar contexts, e.g., [Houlsby, et al. NIPS 2012]: they rewrite the mutual information so that only _marginal_ entropies of y' are required.  These are often much easier to work with.  The new objective is H[y' | x', D] - E[ H[y' | x', D, x*] ], where the expectation is taken over x*.  The former is simply the predictive entropy of a GP, which can be computed in closed form assuming fixed hyperparameters.  The second term is more novel and interesting; we must approximate the predictive uncertainty of y', conditioning the GP on the location of the global maximizer x*.  Conditioning a GP on x* is intriguing, and the bulk of the paper is about an approximate method for doing so.  First, the authors propose a method for sampling from the current distribution p(x* | D).  The proposed method for doing so exploits a now-common trick of approximating the covariance from samples in the Fourier domain.  This gives each sampled function a finite representation, allowing us to find its maximum easily (although the authors omit details on this point).  Given a location x*, they must condition the GP on knowledge of its location.  This would be rather difficult (it essentially requires a censored observation at every point in the domain), so the authors instead propose a set of relaxed but reasonable requirements:* x* must at least be a local maximum.  This is easy to accomplish in the GP framework by conditioning (analytically) on observations forcing the gradient to be zero and the Hessian to be negative definite.  The latter could be difficult in high dimension, so we instead condition H[f] at x* to be diagonal with negative entries.* The value at x* must exceed all those previously seen.* The observed value y' must be less extreme than the value y* we selected subject to the previous constraint.  This reduces the global maximizer condition to just apply at the point x' under consideration, effectively truncating the predictive distribution p(y' | x', D, x*) at y*.The first condition can be analytically handled in the GP framework, but the truncation required by the second is not tractable, so the authors rely on the well-known expectation propagation algorithm for approximating the joint distribution between y* and the diagonal of the Hessian at x*.  Finally, the authors handle uncertainty in the hyperparameters by marginalizing the acquisition function over \theta; this is done by simply slice sampling from p(\theta | D).  A series of experiments indicate that the proposed method (called "predictive" entropy search, but it's really just a new approximation to the same entropy search method) works well.  In particular, Figure 1 is quite convincing at showing that the PES approximation is much closer to the true objective (approximated by rejection sampling) than the ES approximation.  The experiments are well designed and explained, and code will be released upon publication.  Although PES appears to be usually better than ES, it still loses to simpler methods like expected improvement on some problems.One philosophical issue I have with entropy search is that it's not entirely clear to me that H[x* | D] is somehow a "better" objective than, say, EI.  The scenario I always picture is one where we have determined that the optimum of a function must lie in a low, quite flat, but quite broad valley.  In this case we can be very unsure about the exact location of x*, but can be quite certain about its corresponding value y*.  Do I really always need to locate x* with perfect accuracy if I can guarantee that I can find a point with x with f(x) within \epsilon of y* = f(x*)?  Note that a score such as EI can indicate clearly to me that I expect with high certainty not to find a better point and therefore can stop now.  Further, a lookahead version of EI is in fact the Bayesian optimal policy to the optimization problem with simple regret; see [Gruenewaelder, et al. AISTATS 2010].  [Osborne, et al. LION 2009] used a two-step lookahead version of EI that, although prohibitively expensive, encouraged more exploration than the standard, myopic EI criterion and worked better in practice.  I know that this criticism of entropy search is not unique to me, but I have yet to see someone take it head-on and offer a frank discussion on the point.Overall, I like the paper and think the rewriting of the ES objective, together with the interesting and reasonable approximations made by the authors to derive the PES variant, deserve to be published and discussed further.  I also look forward to an eventual code release.Some minor notes:- There are many stacked parens in this paper that are exactly the same height; for an example, see the extreme right-hand-side of (2).  It would help readability (at least for me) if some judicious use of \big, \Big, etc. were used.- Page 1 line 35 (1/35): You say that the derivatives of f are unavailable, but this does not strictly need to be the case.  Indeed you rely on the ease with which we can condition a GP on derivative observations later.  I could imagine a scenario where it might be useful to condition a function on a derivative observation, perhaps by a finite-different approximation as in [Osborne, et al. LION 2009].- 1/52: A comma after "decision" would make this more clear.- 2/80: maxima -> maximum- 6/306: I don't know where the location (0.572, 0.687) is!- 8/426: "hyperparameters" has not been hyphenated until now.