Scalable Hyperparameter Transfer Learning
Bayesian optimization (BO) is a model-based approach for gradient-free black-box function optimization, such as hyperparameter optimization. Typically, BO relies on conventional Gaussian process (GP) regression, whose algorithmic complexity is cubic in the number of evaluations. As a result, GP-based BO cannot leverage large numbers of past function evaluations, for example, to warm-start related BO runs. We propose a multi-task adaptive Bayesian linear regression model for transfer learning in BO, whose complexity is linear in the function evaluations: one Bayesian linear regression model is associated to each black-box function optimization problem (or task), while transfer learning is achieved by coupling the models through a shared deep neural net. Experiments show that the neural net learns a representation suitable for warm-starting the black-box optimization problems and that BO runs can be accelerated when the target black-box function (e.g., validation loss) is learned together with other related signals (e.g., training loss). The proposed method was found to be at least one order of magnitude faster than competing methods recently published in the literature.
This paper proposes a novel Bayesian Optimization approach that is able to do transfer learning across tasks while remaining scalable.  Originality: This is very original work. Bayesian Optimization can work with any probabilistic regression algorithm, so the use of Bayesian linear regression to make it more scalable is well-known, as are its limitations (e.g. it doesn’t extrapolate well). The main novelty here lies in the extension to multi-task learning, which allows it to benefit from prior evaluations on previous tasks. When such evaluations are available, this can provide a significant advantage. The really clever part, however, is to use a neural net to learn a shared representation of the HPO space, and to learn this representation while performing Bayesian Optimization.  Sections 1 and 2 provide a nice overview of related work, which seems quite complete to me.  One very recent related paper that was not mentioned (maybe the authors were unaware of it) is the following arXiv paper: https://arxiv.org/abs/1802.02219 It does transfer learning by storing surrogate models of past tasks, and learns how to weight them in an ensemble during the Bayesian Optimization.  One additional paper that comes to mind (since it also does transfer learning to warm-start BayesOpt) is the following: Wistuba, M. and Schilling, N. and Schmidt-Thieme, L. Learning hyperparameter optimization initializations, DSAA 2015  Clarity: The paper is very well written. It does require solid knowledge of Gaussian Processes to be fully understandable, but otherwise the method and experiments are explained concisely but clearly.  Note: sometimes the explanations are a bit too concise, maybe because of page limits. For instance: L342: “When leveraging the auto-grad operators for the Cholesky” -> Cholesky decomposition.  What is not entirely clear from the paper is how the method learns similarity between tasks/black box functions. Previous work always aims to learn how similar other tasks are, so it can transfer from the right tasks. From my understanding, it seems that the neural net does something similar in that the shared representation captures black-box function similarities. However, I didn’t find this explained very clearly. The results on 30 different OpenML datasets do seem to indicate that it learns similarity. Can this method also tell how similar a new task is to previous tasks, or actually, how similar a its black box function is to previous black box functions?  Significance: This is a very significant contribution. Scalable transfer learning is somewhat of a holy grail in Bayesian optimization, so this paper will very likely have a significant impact, and within the right setting it will likely see significant adoption. Within the scope of the presented experiments, the method seems to work well.  Quality: This is technically a very strong paper. The experiments are nice, although also limited in several regards:  First, the current experiments are a bit limited in that they explore the hyperparameter spaces of single algorithms/functions. These spaces are typically still smallish (3-10 hyperparameters). It would be interesting to see it used on much larger hyperparameter spaces. Other related work, e.g. auto-sklearn with warm-starting, does operate on much larger hyperparameter spaces (e.g. entire pipelines containing a wide range of algorithms and all their hyperparameters).  Second, the evaluation on OpenML datasets compares only to random search and GPs with transfer. It does mention that DNGO and BOHAMIANN were too slow or too unstable. However, it is not clear why other work that uses warm-starting or transfer learning is not compared against, even though this should be relatively easy on OpenML datasets?   It would be good to at least comment on these limitations, or better, add the comparisons.  Update: I grew more worried after reading the author responses. The question on comparing with other scalable HPO techniques was not answered directly. Why not compare against other established and robust and scalable AutoML methods, such as AutoSKLearn/SMAC? You could run that against the same surrogate benchmark? I agree that the methods that are currently compared against are in a way more related, but they also seem not scalable enough. One could argue that the method is interesting enough in itself and that the community can run further comparisons, but sadly there is no code available and the authors did not respond that it will be available after publication. This worries me.