While Bayesian optimization (BO) has achieved great success in optimizing expensive-to-evaluate black-box functions, especially tuning hyperparameters of neural networks, methods such as random search (Li et al., 2016) and multifidelity BO (e.g. Klein et al. (2017)) that exploit cheap approximations, e.g. training on a smaller training data or with fewer iterations, can outperform standard BO approaches that use only full-fidelity observations. In this paper, we propose a novel Bayesian optimization algorithm, the continuous-fidelity knowledge gradient (cfKG) method, that can be used when fidelity is controlled by one or more continuous settings such as training data size and the number of training iterations. cfKG characterizes the value of the information gained by sampling a point at a given fidelity, choosing to sample at the point and fidelity with the largest value per unit cost. Furthermore, cfKG can be generalized, following Wu et al.  (2017), to settings where derivatives are available in the optimization process, e.g. large-scale kernel learning, and where more than one point can be evaluated simultaneously. Numerical experiments show that cfKG outperforms state-of-art algorithms when optimizing synthetic functions, tuning convolutional neural networks (CNNs) on CIFAR-10 and SVHN, and in large-scale kernel learning.
Many black-box optimization problems are "multi-fidelity", in which it
is possible to acquire data with different levels of cost and
associated uncertainty.  The training of machine learning models is a
common example, in which more data and/or more training may lead to
more precise measurements of the quality of a hyperparameter
configuration.  This has previously been referred to as a special case
of "multi-task" Bayesian optimization, in which the tasks can be
constructed to reflect different fidelities.  The present paper
examines this construction with three twists: using the knowledge
gradient acquisition function, using batched function evaluations, and
incorporating derivative observations.  Broadly speaking, the idea is
to allow fidelity to be represented as a point in a hypercube and then
include this hypercube as a covariate in the Gaussian process.  The
knowledge gradient acquisition function then becomes "knowledge
gradient per unit cost" the KG equivalent to the "expected improvement
per unit cost" discussed in Snoek et al (2012), although that paper
did not consider treating fidelity separately.
I don't understand the claim that this is "the first multi-fidelity
algorithm that can leverage gradients".  Can't any Gaussian process
model use gradient observations trivially, as discussed in the
Rasmussen and Williams book?  Why can't any EI or entropy search
method also use gradient observations?  This doesn't usually come up
in hyperparameter optimization, but it seems like a grandiose claim.
Similarly, although I don't know of a paper that explicitly does "A +
B" for multi-fidelity BO and parallel BO, it is an incremental
contribution to combine them, not least because no other parallel BO
methods get evaluated as baselines.
Figure 1 does not make sense to me.  How can the batched algorithm
outperform the sequential algorithm on total cost?  The sequential
cfKG algorithm should always be able to make better decisions with its
remaining budget than 8-cfKG.  Is the answer that "cost" here means
"wall-clock time when parallelism is available"?  If that's the case,
then it is necessary to include plots of parallelized EI, entropy
search, and KG.  The same is true for Figure 2; other parallel BO
algorithms need to appear.