Neural architecture search (NAS) searches architectures automatically for given tasks, e.g., image classification and language modeling. Improving the search efficiency and effectiveness have attracted increasing attention in recent years. However, few efforts have been devoted to understanding the generated architectures. In this paper, we first reveal that existing NAS algorithms (e.g., DARTS, ENAS) tend to favor architectures with wide and shallow cell structures. These favorable architectures consistently achieve fast convergence and are consequently selected by NAS algorithms. Our empirical and theoretical study further confirms that their fast convergence derives from their smooth loss landscape and accurate gradient information. Nonetheless, these architectures may not necessarily lead to better generalization performance compared with other candidate architectures in the same search space, and therefore further improvement is possible by revising existing NAS algorithms.
--- Update after author's response ---
Thank authors for providing the detailed response to all my concerns. I am revising my rating to weak accept.
Specifically, generalising with 52 different variants of depth-width settings are enough, and the updated version is more solid than the earlier one. I think this paper should be accepted.
Summary:
The paper observed one common pattern of searched cell by 5 NAS algorithms, which is the cell found usually has large width but small depth structure, and claims the reason of such pattern is because architectures with shallow but wide structure converge fast during training, and thus are sampled by the NAS policy. To justify this fast convergence claim, the paper proposed to 1) define a width-depth level (width based on feature maps dimension, and depth based on its DAG connection) for each cell in the search space, and randomly sampled one architecture at each level on top of some best-cell discovered by NAS algorithms, 2) training them on original task from scratch independently, and provide visualization of training curve under various learning rate settings, loss landscape as well as gradient variance plot. For theoretical analysis, the paper formulates the narrowest and widest cells, and showing the difference of gradient is bounded by its Lipschitz smoothness of parameter matrices, and usually such variance indicates the widest architecture could converge faster than the narrowest one. 
Whilst this observation is interesting, I found the empirical and theoretical justifications seem to be insufficient to support the claims for the following reasons, 1) The experiments are overwhelmingly built on top of **1** architecture (even obtained from random sampling) of each width-depth level, and it may not well represent the common behavior in the search space, thus the generalization of these claims remains questionable; 2) Theoretical analysis showing the gradient variance difference of narrowest architecture is bounded comparing to the widest cell, however, in practice, these architectures are not properly evaluated. Without proper extension, it is hard to conclude that such difference bound between a wider and narrower architecture pair; 3) experiments in supplementary negatively affects the generalizability of this paper, since the observed trend on DARTS search space does not agree with the one on AmoebaNet and SNAS cases; 4) paper claims the NAS algorithm tends to pick the fast converging architectures more than those late converged one, while intuitive, without showing some detailed process about how NAS algorithms converge, and which architectures they actually sampled during the search phase. 
Nevertheless, I do agree that this paper has a clear motivation, and the observation is interesting and important. If the author could show the conclusion still hold after scaling the experiments, I am learning to accept this paper in the end.
Strength
+ Observation of these NAS algorithms tends to pick wide but shallow cell type is interesting, motivation of this work is well justified.
+ Experiments are throughout, instructive under the paper's current setting.
+ Theoretical justification for the gradient variance between the narrowest and widest cells are sound. 
+ Paper is well written and easy to follow, the figures are presented clearly.
Weakness
- Insufficient experimental setting to support the claim.
My major concern is that under the current experiment design, it is not clear if the observation is well justified, and impedes the main contribution. As mentioned in the review's summary, it is not that convincing that, for each width-depth level, one architecture is enough. I understand exploiting all the variants is resource consuming, however, without such experiments, the current experiments can be impacted by many factors, such as, 1) as in Appendix A2, for each level, the paper "fixed the partial order of their intermediate nodes" and "replace the source node of their associated operations by uniformly randomly sampling a node from their proceeding nodes in the same cell", if I understand correctly, this means the operations will remain the same. However, since these best architectures are searched over both operation and topology connection, new architectures generated from this way may be sub-optimal, hence the larger gradients or slow convergence is not only because they are "narrower" and "deeper". Without proper isolation, it is not possible to conclude as in paper. 
To this end, I suggest author provide the following experiments, 1) sampling all the variants at each level, within a search space like NASBench-101[1], where all the architecture performances are known, 2) at least sampling a sufficient number in the current space (probably > 30 to be statistically significant); 3) Random sampling a small topological variances, then run NAS search algorithms to search the best operation set, then redo the experiments in the paper. 
- Trending of DARTS evaluation results does not agree with SNAS and AmoebaNet.
In Figure  5 (loss landscape plot) and Figure 6 (gradient variance heatmap) for DARTS, the wider-shallower architectures are better comparing to narrower-deeper ones, however, this trend is not significant in Figure 14,16 for AmoebaNet space and Figure 15, 17 for SNAS space. After a closer look, I noticed in Figure 2, the Darts C3 and C4, input node x0 is not connected to the graph, while C, C1, C2, and all the topologies in Figure 10, 11, x0 is connected. Could the author(s) comment on this? Will this be the reason why the C3, C4 DARTS are worse than other architectures?
Minor comments
Page 1 - Introduction paragraph 3 line 1 - 'typologies': is this referring to 'topologies'?
Table 1 - Adapted architectures on CIFAR-10 are mostly worse, even on CIFAR-100, they are better in a small margin. This does not support the claim well.
Figure 9 - Could the author provide the width-depth information for each index? 
Reference
[1] Ying et al. NAS-Bench-101: Towards Reproducible Neural Architecture Search, ICML'19.