HIERARCHICAL REPRESENTATIONS FOR EFFICIENT ARCHITECTURE SEARCH
We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our approach combines a novel hierarchical genetic representation scheme that imitates the modularized design pattern commonly adopted by human experts, and an expressive search space that supports complex topologies. Our algorithm efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which is competitive with the best existing neural architecture search approaches. We also present results using random search, achieving 0.3% less top-1 accuracy on CIFAR-10 and 0.1% less on ImageNet whilst reducing the search time from 36 hours down to 1 hour.
The fundamental contribution of the article is the explicit use of compositionality in the definition of the search space. Instead of merely defining an architecture as a Directed Acyclic Graph (DAG), with nodes corresponding to feature maps and edges to primitive operations, the approach in this paper introduces a hierarchy of architectures of this form. Each level of the hierarchy utilises the existing architectures in the preceding level as candidate operations to be applied in the edges of the DAG. As a result, this would allow the evolutionary search algorithm to design modules which might be then reused in different edges of the DAG corresponding to the final architecture, which is located at the top level in the hierarchy.
Manually designing novel neural architectures is a laborious, time-consuming process. Therefore, exploring new approaches to automatise this task is a problem of great relevance for the field. 
Overall, the paper is well-written, clear in its exposition and technically sound. While some hyperparameter and design choices could perhaps have been justified in greater detail, the paper is mostly self-contained and provides enough information to be reproducible. 
The fundamental contribution of this article, when put into the context of the many recent publications on the topic of automatic neural architecture search, is the introduction of a hierarchy of architectures as a way to build the search space. Compared to existing work, this approach should emphasise modularity, making it easier for the evolutionary search algorithm to discover architectures that extensively reuse simpler blocks as part of the model. Exploiting compositionality in model design is not novel per se (e.g. [1,2]), but it is to the best of my knowledge the first explicit application of this idea in neural architecture search. 
Nevertheless, while the idea behind the proposed approach is definitely interesting, I believe that the experimental results do not provide sufficiently compelling evidence that the resulting method substantially outperforms the non-hierarchical, flat representation of architectures used in other publications. In particular, the results highlighted in Figure 3 and Table 1 seem to indicate that the difference in performance between both paradigms is rather small. Moreover, the performance gap between the flat and hierarchical representations of the search space, as reported in Table 1, remains smaller than the performance gap between the best performing of the approaches proposed in this article and NASNet-A (Zoph et al., 2017), as reported in Tables 2 and 3.
Another concern I have is regarding the definition of the mutation operators in Section 3.1. While not explicitly stated, I assume that all sampling steps are performed uniformly at random (otherwise please clarify it). If that was indeed the case, there is a systematic asymmetry between the probability to add and remove an edge, making the former considerably more likely. This could bias the architectures towards fully-connected DAGs, as indeed seems to occur based on the motifs reported in Appendix A.
Finally, while the main motivation behind neural architecture search is to automatise the design of new models, the approach here presented introduces a non-negligible number of hyperparameters that could potentially have a considerable impact and need to be selected somehow. This includes, for instance, the number of levels in the hierarchy (L), the number of motifs at each level in the hierarchy (M_l), the number of nodes in each graph at each level in the hierarchy (| G^{(l)} |), as well as the set of primitive operations. I believe the paper would be substantially strengthened if the authors explored how robust the resulting approach is with respect to perturbations of these hyperparameters, and/or provided users with a principled approach to select reasonable values.
References:
[1] Grosse, Roger, et al. "Exploiting compositionality to explore a large space of model structures." UAI (2012).
[2] Duvenaud, David, et al. "Structure discovery in nonparametric regression through compositional kernel search." ICML (2013).