One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice. Nevertheless, there is still a lack of understanding on how these weight-sharing algorithms exactly work due to the many factors controlling the dynamics of the process. In order to allow a scientific study of these components, we introduce a general framework for one-shot NAS that can be instantiated to many recently-introduced variants and introduce a general benchmarking framework that draws on the recent large-scale tabular benchmark NAS-Bench-101 for cheap anytime evaluations of one-shot NAS methods. To showcase the framework, we compare several state-of-the-art one-shot NAS methods, examine how sensitive they are to their hyperparameters and how they can be improved by tuning their hyperparameters, and compare their performance to that of blackbox optimizers for NAS-Bench-101.
This paper introduces a benchmarking framework to evaluate a class of one-shot NAS methods by exploiting the NAS-Bench-101 database. In order to do so, the authors apply techniques from Bender et al. (2018) to match the search spaces in NAS-Bench to one-shot scenarios and weight edges in the search graph allowing for a variable number of edges per cell. These weights are then used to query a larger, discrete architecture from NAS-Bench for the corresponding evaluation errors. The authors motivate the soundness of a unique framework by showing minimal differences between three NAS optimizers. 
In their analysis, they also corroborate the findings of Sciuto et al. (2019) on a larger scale, showing little generalization of the ranking from one-shot validation to the final architecture. Finally, the authors also use this framework to show that a fine-tuned baseline compares favorably to SOTA methods, and that large regularization factors lead to more robust configurations.
The paper is well-written and introduces a framework to cheaply compare one-shot NAS optimizers based on expensive computations behind the NAS-Bench-101 benchmark. In addition, the authors provide empirical analyses that reflect generalization and regularization issues with current methods and that could lead towards designing more robust algorithms. 
In my opinion, the the background section might be improved by presenting higher level notions that would lead to an easier understanding by a wider audience. Also, although comprehensible due to the nature of this study, readers not familiar with a variety of previous work might find difficult to parse the paper as many concepts are just referenced.