With the success of deep neural networks, Neural Architecture Search (NAS) as a way of automatic model design has attracted wide attention. As training every child model from scratch is very time-consuming, recent works leverage weight-sharing to speed up the model evaluation procedure. These approaches greatly reduce computation by maintaining a single copy of weights on the super-net and share the weights among every child model. However, weight-sharing has no theoretical guarantee and its impact has not been well studied before. In this paper, we conduct comprehensive experiments to reveal the impact of weight-sharing: (1) The bestperforming models from different runs or even from consecutive epochs within the same run have significant variance; (2) Even with high variance, we can extract valuable information from training the super-net with shared weights; (3) The interference between child models is a main factor that induces high variance; (4) Properly reducing the degree of weight sharing could effectively reduce variance and improve performance.
First of all, I have to state that this is not my area of expertise. So, my review here is an educated guess. 
The paper is an empirical study that looks into the effect of weight sharing in neural network architecture search. The basic idea is that by sharing weights among multiple candidate architectures, the training process can be significantly improved.  In the literature, there have been mixed results, either in favor of or against weight sharing. The question this paper aims to address is to determine if weight sharing is justifiable and to what extent. 
The primary subject investigated by the authors is to determine if the variance of the ranks generated by different runs of the algorithm are highly correlated with each other (e.g. using Kendall rank correlation score). Then, they compared such results with the ground truth (i.e. every child is trained independently). They found that the ranks generated by weight sharing are indeed highly correlated with each other, but there is much larger variance in the ranks when compared to the ground truth method. To understand why this is non-trivial: (1) on one hand, weight sharing speeds up the training process by providing an initial point close to a local minimum, but (2) the local minimum point may or may not  be good for the new architecture.  Hence, one does not know apriori under what conditions would weight sharing be a good strategy. 
The authors also looked into variance of the rank within the same instance by examining how the rank changes with mini-batch epochs. They found that variance is large even within the same instance. 
My primary concern is that the paper is entirely empirical with little if any justification of the results. In addition, it is based on a single architecture and a single dataset. This would have been fine if the results were supported with explanation or theoretical justification. Second, the ultimate goal is to improve the prediction accuracy, not the ranking accuracy. These are not necessarily equivalent. For instance, it is possible that the ranks have a high variance simply because many of the candidate architectures have nearly equivalent performance so the order within them becomes nearly random (and unimportant). In fact, I think the results support this conclusion (see for example Figure 10). Third, some of the highlighted observations are trivial. For example, Observation 1, which states that "Two child models have (higher or lower) interference with each other when they share weights. A child model’s validation accuracy highly depends on the child models it is jointly trained with." I think this observation is trivial. 
Some other comments:
- I would appreciate it if the authors could explain briefly how "prefix sharing" works so that the paper is self-contained. 
- The goal is to help improve the speed of neural architecture search. The authors mention "hints for designing more efficient weight-sharing." Please state those conclusions precisely and clearly. I understand that the authors suggest similarity-based grouping. So, please mention clearly what you recommend in the conclusion section. 
========================== 
#post rebuttal remarks
Thanks for the response. Figure 10 was a typo from my end and I apologize for it. I actually meant figure 3. 
As I said in my review, having an empirical study is acceptable provided that it covers many datasets, not just a single one.