Designing of search space is a critical problem for neural architecture search (NAS) algorithms. We propose a fine-grained search space comprised of atomic blocks, a minimal search unit much smaller than the ones used in recent NAS algorithms. This search space facilitates direct selection of channel numbers and kernel sizes in convolutions. In addition, we propose a resource-aware architecture search algorithm which dynamically selects atomic blocks during training. The algorithm is further accelerated by a dynamic network shrinkage technique. Instead of a search-and-retrain two-stage paradigm, our method can simultaneously search and train the target architecture in an end-to-end manner. Our method achieves state-of-the-art performance under several FLOPs configurations on ImageNet with a negligible searching cost.
[Summary] This paper proposes a channel-wise neural architecture search (NAS) approach. The NAS search algorithm is similar to previous one-shot NAS, but the search space is channel-wise: each channel has it’s own kernel size, which is quite novel and interesting. Results are strong in terms of FLOPS and parameters.
[High-level comments]:
1. I like the novel idea of channel-wise search space, which provides great flexibility for NAS.  Although some recent works (e.g., MixNet) have tried to partition channels into groups, this paper goes further and searches for different kernel size for each single channel.  With this channel-wise search space, it naturally enables a combination of per-channel kernel size selection and per-channel pruning, leading to strong results in terms of FLOPS and parameters, as shown in Figure 4 and Table 1.
2. In general, channel-wise NAS is difficult as different channels are often coupled in various ways. However, the authors observe that recent NAS (such as MnasNet/MixNet/SCARLET-A/EfficientNet) are mostly based on a common MB pattern. By targeting to this specific pattern and applying some additional constraints (e.g. fixed input/output channel size and fixed number of layers per stage as shown in Figure 3), the authors successfully make the channel-wise NAS work well.  I appreciate the authors efforts, but I am also a little concerned that the proposed approach might be limited to this specific small search space. 
[Questions and suggestions to authors]:
3.  How do you justify the generality of the channel-wise search space?  For example, is it possible to also search for input/output channel size (column f in Figure 3) and #layers per stage (column n in Figure 3)? Adding some discussions for this would be very helpful.
4. The title seems too broad. I recommend the authors including “channel-wise” in the title.
5. Please provide some justifications on how to set λ  and c_i in Equation (5).
6. When you say “expands the input channel number from C to 3 × 6C”, what does “3x6C” mean? Is it 18C? What’s the reason for choosing this specific value?
7. Could you show the accuracy and complexity (either FLOPS or params) of the supernet during the training? This information would be helpful to interpret and justify your algorithm 1.
8. The network architecture in Figure 5 is vague. Could you provide the network source code or frozen graph for this specific model?