Strong improvements in neural network performance in vision tasks have resulted from the search of alternative network architectures. Prior work has shown that this search process can be automated and guided by evaluating candidate network performance following limited training (Performance Guided Architecture Search or PGAS). However, because of the large architecture search spaces and the high computational cost associated with evaluating each candidate network, further gains in computational efficiency are needed. Here we present a method termed Teacher Guided Search for Architectures by Generation and Evaluation (TG-SAGE) that produces up to an order of magnitude in search efficiency over PGAS methods. Specifically, TG-SAGE guides each step of the architecture search by evaluating the similarity of internal representations of the candidate networks with those of the (fixed) teacher network. We show that this procedure leads to significant reduction in required per-sample training and that, this advantage holds for two different search spaces of architectures, and two different search algorithms. We further show that in the space of convolutional cells for visual categorization, TG-SAGE finds a cell structure with similar performance as was previously found using other methods but at a total computational cost that is two orders of magnitude lower than Neural Architecture Search (NAS) and more than four times lower than progressive neural architecture search (PNAS). These results suggest that TG-SAGE can be used to accelerate network architecture search in cases where one has access to some or all of the internal representations of a teacher network of interest, such as the brain.
This work tries to accelerate neural architecture search by reducing the computational workload spend to evaluate a single architecture. Given a network with premature performance (i.e. one trained only for few epochs), it tries to compute the mature performance (accuracy achieved when training the network to completion). A score for each architecture is given by "P+TG" which considers the validation accuracy and the similarity of instance representations at different layers to a given teacher network. In an experiment against using the validation accuracy as a score only, they achieve higher validation accuracies on CIFAR-10/100. In a comparison against NASNet and PNASNet, their experiments indicate higher validation accuracy in orders of magnitudes faster run time.
This is a well-written paper with an innovative idea to forecast the mature performance. I am not aware of any other work using the internal representation in order to obtain that. However, there is plenty of other work aiming at accelerating the architecture search by early stopping based on premature performance. Hyperband [1] uses the premature performance (considered in this paper), many others try to forecast the learning curve based on the partial observed learning curve [2,3]. ENAS [4] learns shared parameters for different architectures. Predictions on batches are then used as a surrogate. SMASH [5] uses a hypernetwork to estimate the surrogate. While this work was partly mentioned, neither the close connection to this work is discussed nor serves any of these methods as a baseline.
Typically, the premise of automatic neural architecture search is that the user does not know deep learning well. However, the authors assume that a teacher exist which is an already high performing architecture. It is unclear whether this is a realistic scenario in real-life. How do you find the teacher's architecture in the first place? Does the method also work in cases where the teacher is performing poorly?
The P+TG score depends on a hyperparameter alpha. There is no empirical evidence supporting that alpha can be fixed once and applied for any arbitrary dataset. So I have concerns whether this approach is able to generalize over different datasets. The experiment in the appendix confirms this by showing that a badly chosen alpha will lead to worse performance.
Additionally, I think the paper would benefit from spending more space on explaining the proposed method. I would rather see more details about RDM and maybe a nice plot explaining the motivation rather than repeating NAS and TPE.
[1] Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar: Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. Journal of Machine Learning Research 18: 185:1-185:52 (2017)
[2] Tobias Domhan, Jost Tobias Springenberg, Frank Hutter: Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves. IJCAI 2015: 3460-3468
[3] Bowen Baker, Otkrist Gupta, Ramesh Raskar, Nikhil Naik: Practical Neural Network Performance Prediction for Early Stopping. CoRR abs/1705.10823 (2017)
[4] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, Jeff Dean: Efficient Neural Architecture Search via Parameter Sharing. ICML 2018: 4092-4101
[5] Andrew Brock, Theodore Lim, James M. Ritchie, Nick Weston: SMASH: One-Shot Model Architecture Search through HyperNetworks. CoRR abs/1708.05344 (2017)