Model-Based Relative Entropy Stochastic Search
Stochastic search algorithms are general black-box optimizers. Due to their ease of use and their generality, they have recently also gained a lot of attention in operations research, machine learning and policy search. Yet, these algorithms require a lot of evaluations of the objective, scale poorly with the problem dimension, are affected by highly noisy objective functions and may converge prematurely. To alleviate these problems, we introduce a new surrogate-based stochastic search approach. We learn simple, quadratic surrogate models of the objective function. As the quality of such a quadratic approximation is limited, we do not greedily exploit the learned models. The algorithm can be misled by an inaccurate optimum introduced by the surrogate. Instead, we use information theoretic constraints to bound the 'distance' between the new and old data distribution while maximizing the objective function. Additionally the new method is able to sustain the exploration of the search distribution to avoid premature convergence. We compare our method with state of art black-box optimization methods on standard uni-modal and multi-modal optimization functions, on simulated planar robot tasks and a complex robot ball throwing task. The proposed method considerably outperforms the existing approaches.
Summary: The paper introduces an algorithm (MORE) for black box optimization that constructs local quadratic ``surrogate'' models based on recent observations.By using a (simpler than the true function) quadratic model, the iterative step of refining the reward parameters can be computed in closed form (though the models themselves seem to be built with a form of sampling). This approach allows for a more sample efficient and robust search procedure, which is shown to outperform state-of-the-art methods in terms of samples and converged parameters on a number of simple functions as well as some very complex robotics tasks. Review: The new algorithm performs much better than the state-of-the-art algorithms in a wide range of experiments and is applicable in a very important problem setting.I appreciate the wide range of problems that the authors used and I think they make a very strong case for the new algorithm.I do think there is some room for improvement in what experimental conditions were considered, how some of the results are presented, and in the general description of the algorithm's properties, which I cover here:First and foremost, the experiments section does not contain any results on computational speed and in fact, computation time is not mentioned until the next-to-last sentence in the paper.But there, it is mentioned that MORE takes more time than all the other approaches (though we are told it is fast enough, taking less than a second).It would be better to see some actual comparisons of computation time from the experiments (perhaps in place of one of the robotic arm diagrams), and the computation time issue ought to be mentioned much earlier rather than buried at the very end of the paper.I do not understand why experiments were not conducted using the exact solution of the quadratic modeling problem described in the paragraph starting on line 196.I do understand this would require many more samples than the dimensionality-reduction approach, but it seems possible the exact solution might converge to a better set of parameters.Why not show this approach on some of the graphs (at least in the simple problems) and also report how many more samples it needs than reduced-dimension approach?Or, if the number of samples is always prohibitive, state that definitively to mark the exact solution as intractable.There is a disconnect between the description of the approach in the early sections, which focus on the ability to do closed form calculations, and the computation of the posterior (line 249) of the quadratic models, which requires sampling.I understand that the change in parameters can now be calculated in closed form, which is a nice achievement, but it seems like the burden of sampling may have just been pushed deeper into the algorithm (in the posterior calculation).If this is the case, it should be mentioned earlier on that the algorithm still requires sampling at some level.A basic premise of the paper (line 106) is that keeping the previous and next reward functions close to one another ``controls'' the exploration-exploitation dilemma.But there are certainly cases (say when the current function creates terrible behavior) where the proper thing to do is to completely change the function parameters.A better description around line 106 of when this algorithm is allowed to make such large leaps (or why it does not) would make the approach more understandable.At line 101, it is mentioned that CMA-ES is ``well known as an ad-hoc algorithm'' but no citation is given and no real evidence of that fact is presented.Graphs 4b and 4c seem to be referenced incorrectly in the text (b is referenced as the entropy graph, but its really c, and the reverse for the high dimensional graph)Typos: Line 21 - frail to converge prematurely - converge prematurely. Line 55 -allow to - allow us to Line 107 - search distribution - search distributions Line 110 - control - controlling Line 111 - Such control - Such a control Line 154 - from using - of using Line 155 - decreased the - a decrease in the