Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem. However, DARTS does not work robustly for new problems: we identify a wide range of search spaces for which DARTS yields degenerate architectures with very poor test performance. We study this failure mode and show that, while DARTS successfully minimizes validation loss, the found solutions generalize poorly when they coincide with high validation loss curvature in the architecture space. We show that by adding one of various types of regularization we can robustify DARTS to find solutions with less curvature and better generalization properties. Based on these observations, we propose several simple variations of DARTS that perform substantially more robustly in practice. Our observations are robust across five search spaces on three image classification tasks and also hold for the very different domains of disparity estimation (a dense regression task) and language modelling.
----- Updated after rebuttal period ---
The author's detailed response effectively addressed my concerns. I am moving my score to Accept. This paper proposes an interesting systematic study of differentiable approach in NAS.
------ Original Review ----
Summary 
This paper presents a systematic evaluation on top of differentiable architecture search (DARTS) algorithm and shows it usually searched an architecture with all skip-connection. It empirically reveals that the largest eigenvalue of the Hessian matrix (\lambda) of loss w.r.t. architecture parameters has a strong correlation with the generalization ability (via loss of test dataset), and shows this \lambda will first decrease but then drastically increase after a certain epoch number on 4 different search spaces. It then proposes an early-stop scheme (DARTS-ES) to stop the search before this phenomenon occurs. In addition, it proposes to use data-augmentation, path-dropping and tuning L2 regularization during the search, namely Robust-DARTS(R-DARTS), and yield constantly better results over original DARTS on 3 datasets. 
Overall, the observation that the largest eigenvalue of the Hessian matrix is novel and intriguing, and the experiments are extensive and meaningful. The idea to use more search spaces for comparison is fair and performance increase demonstrates the proposed R-DARTS and DARTS-ES  are effective. Although I still have some questions regarding the detail settings, I think this paper provides a novel angle to understand the search phase of DARTS, and proposed simple but effective regularization can be beneficial to the research community using DARTS. 
Main concerns:
- Problem of DARTS as a motivation 
The claims of local smoothness/sharpness and generalization are related to network generalization is quite intriguing, however, only using largest eigenvalue of Hessian matrix as an indicator of this local shape does not seem to be enough. A recent paper on loss-landscape visualization [1] provides means to examine this hypothesis directly, and could the author try to provide additional visualization to support their claim? Otherwise, the paper's claim does not generalize to the local shape of the loss function, and should stays with the largest eigenvalue. It is totally okay in my perspective, but just indicates some revision to the main text and analysis of Section 4.2.
- Questions about Figure 3 experiments
How is test error computed? Is it on a batch of test-split, or the entire one? Also, which architecture is used to compute this test error? Paper mentioned, in Section 4.1, the word "final architecture", but does this refer to the super-net (the one-shot model in paper's definition), or the stand-alone model obtained via binarized architecture alphas? If latter, is this generalization error obtained from training from scratch? Or simply using the super-net parameters during the search? Since the conclusion of this plot serves as the foundation of designing R-DARTS and DARTS-ES, if the experiments are only conducted over a small set of images or the binarized model with super-net parameters, it undercuts the credibility of the conclusion, largest architectural eigenvalue, and generalization ability.
- More independent runs of experiments.
In Figure 3, validation of original DARTS, and Table 1, DARTS vs DARTS-ES, paper runs the experiment for 3 times and take the average, but for the proposed R-DARTS, it is not. Is there a reason why not scaling the experiments? I suggest the author provide results over 5 runs, like the one in Table 4 for PTB and show if the R-DARTS truly surpasses DARTS constantly. This question also applies to Figure 1, when paper claims the original DARTS found poor cell type, could this be repetitive over multiple runs?
I guess for the experiments in Table 3, it is already done since paper mentioned the reported results are the best model of searching 4 times. 
- R-DARTS failed to out-perform DARTS in the original space on CIFAR-10
This is confusing, will this suggest, if tuning well, DARTS will surpass R-DARTS(L2) in other cases as well? Since this is the only setting that DARTS is built upon. 
Minor comments and questions
- Using test data during search
After showing the strong correlation between the largest eigenvalue of Hessian and the network generalization error, the early-stop is natural, however, does this mean the model selection is using the test data? Or the actual test-data is never seen during the search phase of Section 4.3.
- L2 stabilizes max eigenvalue
Paper uses L2 coefficient up to 0.0243, showing constant improvement of test error while validation error drops in CIFAR-10 of Figure 11. Could the author try larger coefficients to determine when this trend will stop? 
- Question about section 4.2
Performance drop due to the binarized operation (pruning step) in DARTS analysis is very interesting, I am curious how many architectures does the paper evaluate in Figure 5, when the dominant eigenvalue is smaller than 0.25? Since the conclusion is "low curvature never led to large performance drops" if the number of points is too few, it is not that convincing, especially from the plot, we see at eigenvalue = 0.5, there exists 2 architecture with >20% drop. In addition, what does each point in Figure 5 refer to? The best model (and the binarized one according to the argmax of \alpha) of one independent DARTS run or some binarized models sampled from a distribution on top of the same DARTS run (meaning only one super-net)? Is this experiment follows the setting in Figure 4? 
- Figure 6, C10 S2, DARTS-ES is worse than DARTS when Drop probability = 0.6, whereas all other cases, DARTS-ES outperforms DARTS, why does this happen? Could the author comment on it?
- ScheduledDropPath in section 5.1
Does Drop-path belongs to data-augmentation techniques? It is more like a regularization in my perspective and should be grouped with 5.2.
- S1 S2... in Table 3
Does this refer to the search space? Or different random seed (mentioned )
- one-shot v.s. weight sharing model
One-shot in NAS domain is firstly introduced by Bender et al., while Pham et al. use parameter sharing. The reason to use one-shot is that all the sub-paths will have a fair chance to be trained. 
- Typos
1. In section 2.1, line 4 "better.Similarly" should have space.
--- Reference ---
[1] Li et al., Visualizing the Loss Landscape of Neural Nets, arxiv'17.