We revisit the one-shot Neural Architecture Search (NAS) paradigm and analyze its advantages over existing NAS approaches. Existing one-shot method (Bender et al., 2018), however, is hard to train and not yet effective on large scale datasets like ImageNet. This work propose a Single Path One-Shot model to address the challenge in the training. Our central idea is to construct a simplified supernet, where all architectures are single paths so that weight co-adaption problem is alleviated. Training is performed by uniform path sampling. All architectures (and their weights) are trained fully and equally. Comprehensive experiments verify that our approach is flexible and effective. It is easy to train and fast to search. It effortlessly supports complex search spaces (e.g., building blocks, channel, mixed-precision quantization) and different search constraints (e.g., FLOPs, latency). It is thus convenient to use for various needs. It achieves start-of-the-art performance on the large dataset ImageNet.
This paper presents a new one-shot NAS approach. Parameter updating and structure updating are optimized separately. In the process of parameter optimization, different from the previous methods, the network samples the structure according to the uniform probability distribution. This sampling method can avoid the coupling caused by optimizing the structural parameters at the same time. After training the supernet, the network uses the genetic algorithm to search the structure.
Pros:
+ This paper is well-written and easy to follow. 
+ The method proposed in this paper is very efficient in the search stage and saves memory. During the searching stage, the constraints of the model can be restricted by the genetic algorithm. 
+ This method could directly search architectures on the large scale dataset.
+ The results are promising. Experiments are performed on various structures and including quantization layers. 
Still, I have several minor concerns regarding the algorithm and experiments.
1. Why does the author think that the supernet needs to be trained by single path all the time? In the architecture searching space, under the same computational constraints, some models perform well, and some do not. Why do the good models and the worse models use the same probability to sample? Wouldn't it require much more time to optimize the supernet? In the experiment part, the author compares a related work FBNet, which uses the structure parameter \theta and temperature parameter \tau to control the sampling. When the temperature \tau is high, the distribution of the samples degenerates into the single path sampler, and the accuracy is used to optimize the parameter \theta, which makes \theta tend to select the well-performed models. With the decrease of the temperature \tau, the probability of sampling well-performed architectures is higher than the worse-performed models. Therefore, what is the advantage of the single path sampling method over \tau controlled sampler? It seems that gradually pruning the worse-performed search space would accelerate searching time. More analysis is preferred rather than the superior results from the experiments.
2. Is identity in the search space? For at the last of Page4, the author said 'our choice block does not have an identity branch', and the listed architecture on page 13 has the identity.
3. Is the supernet training for 120 epochs necessary? Is the rank of network structures become steady when training for 120 epochs. Because for the subsequent EA-based network structure search, the loss scale is not important, but the sorting (ranking) is important. (This beyond the scope of this paper, but relates to the NAS area)
4. What is the number of parameters for all the listed models in Tab. 3 & Tab. 4? Listing the parameters makes it easier for the followers.
5. What kind of structure do the mixed-precision networks use? The original resnet module, or the bireal resnet module [r1]?
6. While recalculating the statistics of all the BN,  is backpropagation required or just run the inference without gradient backpropagation.
7. The author uses the approximate complete set of Imagenet to train supernet. If we directly inherit the parameter from the supernet, can we accelerate the training of searched good structure? Will Top-1 acc lower than training from scratch?
[r1] Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm