Neural Architecture Search (NAS) has emerged as a promising technique for automatic neural network design. However, existing NAS approaches often utilize manually designed action space, which is not directly related to the performance metric to be optimized (e.g., accuracy). As a result, using manually designed action space to perform NAS often leads to sample-inefficient explorations of architectures and thus can be sub-optimal. In order to improve the sample efficiency, this paper proposes Latent Action Neural Architecture Search (LaNAS), which learns actions to recursively partition the search space into good or bad regions that contain networks with concentrated performance metrics, i.e., low variance. During the search phase, as different architecture search action sequences lead to regions with different performance, the search efficiency can be significantly improved by biasing towards the good regions. On 3 NAS datasets, our experimental results demonstrated that LaNAS is 37×, 21.7×, 19.3×, 10.7×, 12.3×, 16.5× more sample-efficient than Random Search, Regularized Evolution, Monte Carlo Tree Search, Neural Architecture Optimization, TPE and SMAC, respectively. When applied to the open domain, LaNAS achieves 98.0% accuracy on CIFAR-10 and 75.0% top1 accuracy on ImageNet in only 803 samples, outperforming SOTA AmoebaNet with 33× fewer samples.
This paper introduces a Neural Architecture Search algorithm that attempts to solve the problems of the existing NAS only utilizing manually designed action space (not related to the performance). The paper proposes LaNAS which is based on an MCTS algorithm to partition the search space into tree nodes by the performance in the tree structure. The performance of the method is shown in the NASBench-101 dataset and Cifar-10 open domain search.
I lean to reject this paper because (1) the motivation is not well justified by the experiments, (2) the comparison on NASBench-101 is not convincing, (3) some important explanation of the method is missing.
Main arguments
The main contribution of the paper is using a learned action space in MCTS rather than a manually designed MCTS algorithm for NAS. However, as far as I know, the MCTS approach for the NAS problem is not a standard solution for NAS (which is not proved to be practically useful in other people’s papers) which diminishes the contribution of the improvements of MCTS in NAS.
Lack of the main comparison. For the motivation of the proposed method, the authors mention the drawbacks of other NAS methods used fixed action space in their RL or MCTS module. However, the authors only show that using a learned action space in MCTS is better than a fixed MCTS algorithm in the experiments. What about using a learned action space in the RL module such as PPO in NAS comparing to the fixed one?
The comparison of NASBench-101 is not convincing. The authors compared with the BO method and claimed that the method is 16.5x more efficient than BO. However, the recently released paper “BANANAS: Bayesian Optimization with Neural Networks for Neural Architecture Search” said that their method is 3.8x more efficient than the one proposed here, which is quite confusing.
Some important explanation of the methods is missing. Throughout the paper, the method to sample from a leaf node is only mentioned in 3.3. However, the corresponding sampling method is unclear. The paper only mentions that MCMC has been adopted to sample from the target subspace. In my opinion, it is not so trivial to use MCMC here and should be elaborated in more detail. Otherwise, people cannot use it.
As given in figure 3, if c is set to be a very small number, the search is similar to simply using a series of predictors and always samples the models with better-predicted accuracies. Is MCTS really useful here? To show the effectiveness of MCTS, it is recommended to experiment on different values of c.
Results given in the upper row of Fig.5 is not useful. In practice, it is already painful to sample about 1000 models and train them for the Cifar-10 dataset. It is more useful to see how this method behaves in the range of (0,1000). However, in Fig.5, different methods are all overlapping in this range and hard to tell whether this method is better than other methods