There is growing interest in automating designing good neural network architectures. The NAS methods proposed recently have significantly reduced architecture search cost by sharing parameters, but there is still a challenging problem of designing search space. We consider search space is typically defined with its shape and a set of operations and propose a channel-level architecture search (CNAS) method using only a fixed type of operation. The resulting architecture is sparse in terms of channel and has different topology at different cell. The experimental results for CIFAR-10 and ImageNet show that a fine-granular and sparse model searched by CNAS achieves very competitive performance with dense models searched by the existing methods.
This paper aims to search a sparse but competitive architecture with using a single fixed type of operation by proposing a channel-level neural architecture search (CNAS). Different from most previous NAS works, this paper conducts NAS process on channel-level such that different cell has different topology. CNAS provides a heuristic algorithm to calculate the saliency vector and zero out the channels iteratively until satisfying a given sparsity. This paper performs CNAS on Cifar-10 and ImageNet, and analyzes the topological properties of the final model. The results of experiment demonstrate CNAS can reach a competitive model with dense models searched by baselines. 
This paper provides us with a novel insight that searching neural architecture on the channel level instead of operation and connection level. However, it just combines NAS and pruning parts together, which lacks of novelty in the algorithm level.
I lean to reject this paper because: (1) it lacks of novelty, (2) the experiment result is not convincing, (3) some related works are missed, (4) the expression of the paper is not clear.
Main argument
CNAS is a straightforward combination of NAS and pruning. As the author said in the section 2.1, CNAS method can be seen as two separate processes: training a supernet like one-shot NAS methods and then conducting pruning on the found supernet using Taylor expansion criteria. Both parts are the same as previous works almost and there is no innovation and improvements.
Many related works are missed in the paper. One important step in CNAS is pruning, which uses Taylor expansion technic as previous work. However, it only introduces NAS in introduction section and related works section, ignoring the pruning process. From my view, the pruning part is more important than NAS part.
From the results of the experiment, the improvement of CNAS is not convincing. I think the main focus of the paper is the sparsity, but in Table 2, the number of parameters of model is still larger (4.6*10^6) compared with some baselines like DARTS (2.9*10^6). Besides that, much space in the experiment section is devoted to the relationship between supernet and the final model, which is not so important. Because in other methods, supernet is just an intermedia. Therefore, The comparison between them is not so meaningful.
The paper is hard to understand because of unclear writing. For example, in algorithm part, the author doesn't make DimensionReduction function and its inputs clear. The author mentions the first input in the paragraph but how to combine with the second input? Also, the representation in Figure 1(b) is confused. It's hard to figure out "the thick edges", "the solid thick ones" and "the dotted solid thick ones".
Questions
1. As we all known, operation set is important to the search space. Have you tried more types of operations? From my view, using only one fixed operation is unfair for CNAS compared with other methods.
2. One of your focus is sparsity of the model. Can you explain the reason that you set the number of parameters to a large value (4.6*10^6)? Have you tried to use a larger sparsity value? What's the performance of CNAS when the model is sparser?
 
3. In Table 2, there are some different tricks (cutout, autoaugmented) applied on some methods. Can you explain how you guarantee the fair comparison between different methods? If we just compare CNAS-R or CNAS-W, they are not better than baselines.