Neural Architecture Search (NAS) aims at automatically finding neural network architectures within an enormous designed search space. The search space usually contains billions of network architectures which causes extremely expensive computing costs in searching for the best-performing architecture. One-shot and gradient-based NAS approaches have recently shown to achieve superior results on various computer vision tasks such as image recognition. With the weight sharing mechanism, these methods lead to efficient model search. Despite their success, however, current sampling methods are either fixed or hand-crafted and thus ineffective. In this paper, we propose a learnable sampling module based on variational auto-encoder (VAE) for neural architecture search (NAS), named as VAE-NAS, which can be easily embedded into existing weight sharing NAS framework, e.g., one-shot approach and gradient-based approach, and significantly improve the performance of searching results. VAENAS generates a series of competitive results on CIFAR-10 and ImageNet in NasNet-like search space. Moreover, combined with one-shot approach, our method achieves a new state-of-the-art result for ImageNet classification model under 400M FLOPs with 77.4% in ShuffleNetlike search space. Finally, we conduct a thorough analysis of VAENAS on NASbench-101 dataset, which demonstrates the effectiveness of our proposed methods.
============ comments after rebuttal
I would like to thank the authors for addressing some of my concerns. I believe the new results under Q2 and Q3 are useful additions to strengthen the paper.
As for the authors' comments for Q1, I'd like to point out that a "larger" search space is not necessarily more difficult (a more meaningful metric would be the average accuracy of random architectures). It is still possible that the current search space is putting the proposed method at advantage, especially if having dense connections is a useful prior.
Given the above, I would like to increase my score from 1 to 3 (weak reject).
============ previous comments
Neural architecture search can be formulated as learning a distribution of promising architectures (the sampling policy). Such a distribution is usually represented in a fully factorized fashion (e.g., as a set of multinational distributions as in DARTS). This paper proposes to model the architecture distribution using a VAE instead, where the encoder and decoder are implemented using LSTMs. The authors argue that the increased flexibility of the sampling policy leads to improved performance on CIFAR-10, NASBench and ImageNet.
The idea of representing the architecture distribution using VAEs is very natural, which in principle could offer better coverage over interesting regions in the search space as compared to traditional factorized distribution representation (which has a single mode only).
While the method itself is interesting, I do not think it has been properly backed up by controlled experiments. This is largely due to the fact that the authors are comparing their method against baselines in fundamentally different search spaces. For instance:
* For CIFAR-10 experiments, the authors mentioned in the appendix: "Different from DARTS, in our search space, one node could have more than two predecessors in one cell". This makes the search space very different from the existing ones as used by NASNet/AmoebaNet/DARTS/SNAS, and it hence remains unclear to what degree the resulting architecture has benefited from the increased in-degrees per node. Note the searched densely connected cells in Figure 4 & 5 in Appendix A.4 are clearly not part of the search space for many of the baselines.
* For ImageNet experiments, the authors are using a ShuffleNet-like search space which has fundamentally different building blocks than other architecture search baselines (commonly built on top of inverted bottleneck layers). It is unclear to what degree the 77.4 top-1 accuracy @ 365 MFlops results have benefited from this different search space.
Without fair comparisons in a controlled setup, it is impossible for readers to draw any solid conclusion about the true empirical advantages of the method. I'm therefore unable to recommend acceptance for the paper at the moment, but am willing to raise my score if the authors can properly address those issues in the rebuttal.
Additional question: How can we isolate it to tell whether the gains come from the LSTMs or the VAEs? Is there any intuition why incorporating a generative sampler based on VAEs is potentially superior to method like ENAS (which involves LSTMs decoders only)?