HYPERPARAMETER OPTIMIZATION: A SPECTRAL APPROACH
We give a simple, fast algorithm for hyperparameter optimization inspired by techniques from the analysis of Boolean functions. We focus on the high-dimensional regime where the canonical example is training a neural network with a large number of hyperparameters. The algorithm -an iterative application of compressed sensing techniques for orthogonal polynomials -requires only uniform sampling of the hyperparameters and is thus easily parallelizable. Experiments for training deep neural networks on Cifar-10 show that compared to state-of-the-art tools (e.g., Hyperband and Spearmint), our algorithm finds significantly improved solutions, in some cases better than what is attainable by handtuning. In terms of overall running time (i.e., time required to sample various settings of hyperparameters plus additional computation time), we are at least an order of magnitude faster than Hyperband and Bayesian Optimization. We also outperform Random Search 8×. Our method is inspired by provably-efficient algorithms for learning decision trees using the discrete Fourier transform. We obtain improved sample-complexty bounds for learning decision trees while matching state-of-the-art bounds on running time (polynomial and quasipolynomial, respectively).
The paper is about hyperparameter optimization, which is an important problem in deep learning due to the large number of hyperparameters in contemporary model architectures and optimization algorithms.
At a high-level, hyperparameter optimization (for the challenging case of discrete variables) can be seen as a black-box optimization problem where we have only access to a function evaluation oracle (but no gradients etc.). In the entirely unstructured case, there are strong lower bounds with an exponential dependence on the number of hyperparameters. In order to sidestep these impossibility results, the current paper assumes structure in the unknown function mapping hyperparameters to classification accuracy. In particular, the authors assume that the function admits a representation as a sparse and low-degree polynomial. While the authors do not empirically validate whether this is a good model of the unknown function, it appears to be a reasonable assumption (the authors *do* empirically validate their overall approach).
Based on the sparse and low-degree assumption, the paper introduces a new algorithm (called Harmonica) for hyperparameter optimization. The main idea is to leverage results from compressed sensing in order to recover the sparse and low-degree function from a small number of measurements (i.e., function evaluations). The authors derive relevant sample complexity results for their approach. Moreover, the method also yields new algorithms for learning decision trees.
In addition to the theoretical results , the authors conduct a detailed study of their algorithm on CIFAR10. They compare to relevant recent work in hyperparameter optimization (Bayesian optimization, random search, bandit algorithms) and find that their method significantly improves over prior work. The best parameters found by Harmonica improve over the hand-tuned results for their "base architecture" (ResNets).
Overall, I find the main idea of the paper very interesting and well executed, both on the theoretical and empirical side. Hence I strongly recommend accepting this paper.
Small comments and questions:
1. It would be interesting to see how close the hyperparameter function is to a low-degree and sparse polynomial (e.g., MSE of the best fit).
2. A comparison without dummy parameters would be interesting to investigate the performance differences between the algorithms in a lower-dimensional problem.
3. The current paper does not mention the related work on hyperparameter optimization using reinforcement learning techniques (e.g., Zoph & Le, ICLR 2017). While it might be hard to compare to this approach directly in experiments, it would still be good to mention this work and discuss how it relates to the current paper.
4. Did the authors tune the hyperparameters directly using the CIFAR10 test accuracy? Would it make sense to use a slightly smaller training set and to hold out say 5k images for hyperparameter evaluation before making the final accuracy evaluation on the test set? The current approach could be prone to overfitting.
5. While random search does not explicitly exploit any structure in the unknown function, it can still implicitly utilize smoothness or other benign properties of the hyperparameter space. It might be worth adding this in the discussion of the related work.
6. Algorithm 1: Why is the argmin for g_i  (what does the index i refer to)?
7. Why does PSR truncate the indices in alpha? At least in "standard" compressed sensing, the Lasso also has recovery guarantees without truncation (and empirically works sometimes better without).
9. Definition 3: Should C be a class of functions mapping {-1, 1}^n to R?  (Note the superscript.)
10. On Page 3 we assume that K = 1, but Theorem 6 still maintains a dependence on K. It might be cleaner to either treat the general K case throughout, or state the theorem for K = 1.
11. On CIFAR10, the best hyperparameters do not improve over the state of the art with other models (e.g., a wide ResNet). It could be interesting to run Harmonica in the regime where it might improve over the best known models for CIFAR10.
12. Similarly, it would be interesting to see whether the hyperparameters identified by Harmonica carry over to give better performance on ImageNet. The authors claim in C.3 that the hyperparameters identified by Harmonica generalize from small networks to large networks. Testing whether the hyperparameters also generalize from a smaller to a larger dataset would be relevant as well.