  In the literature on hyper-parameter tuning, a number of recent solutions
rely on low-fidelity observations (e.g., training with sub-sampled datasets) in
order to efficiently identify promising configurations to be then tested via
high-fidelity observations (e.g., using the full dataset). Among these,
HyperBand is arguably one of the most popular solutions, due to its efficiency
and theoretically provable robustness. In this work, we introduce HyperJump, a
new approach that builds on HyperBand's robust search strategy and complements
it with novel model-based risk analysis techniques that accelerate the search
by skipping the evaluation of low risk configurations, i.e., configurations
that are likely to be eventually discarded by HyperBand. We evaluate HyperJump
on a suite of hyper-parameter optimization problems and show that it provides
over one-order of magnitude speed-ups, both in sequential and parallel
deployments, on a variety of deep-learning, kernel-based learning, and neural
architectural search problems when compared to HyperBand and to several
state-of-the-art optimizers.

Machine Learning