Recently Neural Architecture Search (NAS) has aroused great interest in both academia and industry, however it remains challenging because of its huge and non-continuous search space. Instead of applying evolutionary algorithm or reinforcement learning as previous works, this paper proposes a Direct Sparse Optimization NAS (DSO-NAS) method. In DSO-NAS, we provide a novel model pruning view to NAS problem. Specifically, we start from a completely connected block, and then introduce scaling factors to scale the information flow between operations. Next, we impose sparse regularizations to prune useless connections in the architecture. Lastly, we derive an efficient and theoretically sound optimization method to solve it. Our method enjoys both advantages of differentiability and efficiency, therefore can be directly applied to large datasets like ImageNet. Particularly, on CIFAR-10 dataset, DSO-NAS achieves an average test error 2.84%, while on the ImageNet dataset DSO-NAS achieves 25.4% test error under 600M FLOPs with 8 GPUs in 18 hours.
Summary:
This paper proposes Direct Sparse Optimization (DSO)-NAS, which is a method to obtain neural architectures on specific problems, at a reasonable computational cost.
The main idea is to treat all architectures as a Directed Acyclic Graph (DAG), where each architecture is realized by a subgraph. All architectures in the search space thus share their weights, like ENAS (Pham et al 2018) and DARTS (Liu et al 2018a). The DAG’s edges can be pruned via a sparsity regularization term. The optimization objective of DSO-NAS is thus:
Accuracy + L2-regularization(W) + L1-sparsity(\lambda),
where W is the shared weights and \lambda specifies which edges in the DAG are used.
There are 3 phases of optimization:
1. All edges are activated and the shared weights W are trained using normal SGD. Note that this step does not involve \lambda.
2. \lambda is trained using Accelerated Proximal Gradient (APG, Huang and Wang 2018).
3. The best architecture is selected and retrained from scratch.
This procedure works for all architectures and objectives. However, DSO-NAS further proposes to incorporate the computation expense of architectures into step (2) above, leading to their found architectures having fewer parameters and a smaller FLOP counts.
Their experiments confirm all the hypotheses (DSO-NAS can find architectures, having small FLOP counts, having good performances on CIFAR-10 and ImageNet).
Strengths:
1. Regularization by sparsity is a neat idea.
2. The authors claim to be the first NAS algorithm to perform direct search on ImageNet. Honestly, I cannot confirm this claim (not sure if I have seen all NAS papers out there), but if it is the case, then it is impressive.
3. Incorporating architecture costs into the search objective is nice. However, this contribution seems to be orthogonal to the sparsity regularization, which, I suppose, is the main point of the paper.
Weaknesses:
1. Some experimental details are missing. I’m going to list them here:
- Was the auxiliary tower used during the training of the shared weights W?
- Figure 4 does not illustrate M=4 and N=4, e.g. which operation belongs to which layer?
- Did the experiments on CIFAR-10 and ImageNet use the cosine learning rate schedule [1]? If or if not, either way, you should specify it in a revised version of this paper, e.g. did you use the cosine schedule in the first 120 steps to train the shared parameters W, did you use it in the retraining from scratch?
- In Section 3.3, it is written that “The sparse regularization of \lambda induces great difficulties in optimization”. This triggers my curiosity of which difficulty is it? It would be nice to see this point more elaborated, and to see ablation study experiments.
2. Missed citation: MnasNet [2] also incorporates the cost of architectures in their search process. On ImageNet, your performance is similar to theirs. I think this will be a good comparison.
3. The paper has some grammatical errors. I obviously missed many, but here are the one I found:
- Section 3.3: “Different from pruning, which the search space is usually quite limited”. “which” should be “whose”?
- Section 4.4.1: “DSO-NAS can also search architecture [...]”  -> “DSO-NAS can also search for architectures [...]”
References.
[1] SGDR: Stochastic Gradient Descent with Warm Restarts. https://arxiv.org/pdf/1608.03983.pdf
[2] MnasNet: Platform-Aware Neural Architecture Search for Mobile. https://arxiv.org/pdf/1807.11626.pdf