Multi-Information Source Optimization
We consider Bayesian methods for multi-information source optimization (MISO), in which we seek to optimize an expensive-to-evaluate black-box objective function while also accessing cheaper but biased and noisy approximations ("information sources"). We present a novel algorithm that outperforms the state of the art for this problem by using a Gaussian process covariance kernel better suited to MISO than those used by previous approaches, and an acquisition function based on a one-step optimality analysis supported by efficient parallelization. We also provide a novel technique to guarantee the asymptotic quality of the solution provided by this algorithm. Experimental evaluations demonstrate that this algorithm consistently finds designs of higher value at less cost than previous approaches.
This paper deals with the important topic of optimization in cases where in addition to costly evaluations of the objective function, it is possible to evaluate cheaper approximations of it. This framework is referred to as MISO (multi-information source optimization) in the paper, where Bayesian optimization strategies relying on Gaussian process models are considered. An original MISO approach is presented, misoKG, that relies on an adaption of the Knowledge Gradient algorithm in multi-information source optimization settings. The method is shown to achieve 
very good results and to outperform considered competitors on three test cases.
Overall, I am fond of the ideas and the research directions tackled by the paper, and I found the contributions principled and practically relevant. Given the potential benefits for the machine learning community and beyond, these contributions should definitely be published in a high-level venue such as NIPS. Yet, the paper could still be improved in several respects. Le me start by general remarks and then add minor comments.
A lot of emphasis is put throughout the paper, and starting in the abstract, on the use of "a novel Gaussian process covariance kernel". In the introduction, "two innovations" are listed as 1) the kernel and 2) the acquisition function. I think that this is misleading, and to me the main contributions concern by far the acquisition function in the first place, and the model is more a modality of application. By the way, the proposed model is a quite simple particular case 
of the Linear Model of Corregionalization (which the authors mention), corresponding to an M*(M+1) matrix [1,I], i.e. a concatenation of a vector of ones with an identity matrix. Besides this, it also has quite some similarity with the Kennedy and O'Hagan model that was used by Forrester et al. in optimization, without neither the hierarchical aspect nor regression coefficients. In all, I have the feeling that using the model presented by the authors is indeed a clever thing to do, but to me it doesn't constitute a major contribution in itself. Not also that if misoKG is compared to several competitors, I did not see comparisons of the proposed model (in terms of fit) versus others. Also, what would prevent from using other criteria with this model or other models with the misoKG aquisition function? 
Other than that, let me formulate a few comments/questions regarding the main contributions presented in the third section. First, a finite set of points is 
introduced at each iteration for the calculation and the optimization of the acquisition function. I am wondering to what extent such a set is needed on the one 
hand for the calculation, on the other hand for the optimization, and finally if there are good reasons (beyond the argument of simplicity) why the same 
set should be used for both tasks. I conceive that optimizing over l and x at the same time is uneasy, but couldn't one optimize over D for every l and then 
compare the solutions obtained for the different l's? By the way, an approach was proposed in "Bayesian Optimization with Gradients" (arXiv:1703.04389) to 
avoid discretization in the calculation of a similar acquisition function; could such approach be used here? If yes but a discretization approach is preferred 
here, maybe a discussion on when considering discrete subsets anyway would make sense? On a different note, I was wondering to what extent the calculation tricks 
used around lines 158-172 differ from the ones presented in the PhD thesis of P. Frazier and related publications. Maybe a note about that would be a nice addition. 
Finally, I found the formulation of Theorem 1 slightly confusing. One speaks of a recommendation x_N^star in A and then take limit over N. So we have a sequence of 
points in A, right? Besides, it all relies on Corollary 1 from the appendix, but there the reader is referred to "the full version of the paper"...I do not have 
that in my possession, and as of now I found the proof of Corollary 1 somehow sketchy. I can understand that there are good reasons for that, but it is bit strange 
to comment on a result which proof is in appendix, which itself refers to a full version one doesn't have at hand!
A few minor comments follow: 
* About the term "biased" (appearing in several places): I understand well that a zero-mean prior on the bias doesn't mean at all that there is no bias, but 
maybe manipulating means of biases without further discussion on the distinction between the bias and its model could induce confusion. Also, around lines 54-55, 
it is written that "...IS may be noisy, but are assumed to be unbiased" followed by "IS are truly unbiased in some applications" and "Thus, multi-fidelity methods 
can suffer limited applicability if we are only willing to use them when IS are truly unbiased...". As far as I understand, several methods including the ones 
based on the Kennedy and O'Hagan settings allow accounting for discrepances in the expectations of the different IS considered. So what is so different between 
multi-fidelity optimization and what is presented here regarding the account of possible "biases"? From a different perspective, other approaches like the one of 
"Quantile-Based Optimization of Noisy Computer Experiments with Tunable Precision" are indeed relying on responses with common expectation (at any given x).
* In line 43, the "entropy" is the maximizer's entropy, right? 
* Still in line 43: why would the one-step optimality mean that the proposed approach should outperform entropy-based ones over a longer horizon? 
* About "From this literature, we leverage...", lines 70-72: as discussed above, it would be wothwhile to point out more explicitly what is new versus what is 
inherited from previous work regarding these computational methods. 
* In line 79 and in the bibliography: the correct last name is "Le Gratiet" (not "Gratiet"). 
* In line 84: without further precision, argmax is a set. So, the mentioned "best design" should be one element of this set? 
* In line 87: the "independence" is a conditional independence given g, I guess...? 
* In page 3 (several instances): not clear if the bias is g(.)-f(l,.) or delta_l(.)=f(l,.)-g(.)?
* In line 159, the first max in the expectations should have braces or so around a_i + b_i Z. 
* In line 165: using c is a bit slippery as this letter is already used for the evaluation costs...
* In "The Parallel Algorithm": strange ordering between 1) and 2) as 2) seems to describe a step within 1)...?