GRAPH HYPERNETWORKS FOR NEURAL ARCHITECTURE SEARCH
Neural architecture search (NAS) automatically finds the best task-specific neural network topology, outperforming many manual architecture designs. However, it can be prohibitively expensive as the search requires training thousands of different networks, while each can last for hours. In this work, we propose the Graph HyperNetwork (GHN) to amortize the search cost: given an architecture, it directly generates the weights by running inference on a graph neural network. GHNs model the topology of an architecture and therefore can predict network performance more accurately than regular hypernetworks and premature early stopping. To perform NAS, we randomly sample architectures and use the validation accuracy of networks with GHN generated weights as the surrogate search signal. GHNs are fast -they can search nearly 10× faster than other random search methods on CIFAR-10 and ImageNet. GHNs can be further extended to the anytime prediction setting, where they have found networks with better speed-accuracy tradeoff than the state-of-the-art manual designs.
This paper proposes to accelerate architecture search by replacing the expensive inner loop (wherein candidate architectures are trained to completion) with a HyperNetwork which predicts the weights of candidate architectures, as in SMASH. Contrary to SMASH, this work employs a Graph neural network to allow for the use of any feedforward architecture, enabling fast architecture search through parameter prediction using highly performant search spaces. The authors test their system and show that performance using Graph HyperNet-generated weights correlates with performance when trained normally. The authors benchmark their method against competing approaches ("traditional" NAS techniques which incur the full expense of the inner loop, and one-shot techniques which learn a large model then select architectures by searching for paths in said model) and show competitive performance.
This is a solid technical contribution with a well-designed set of experiments. While the novelty is not especially high, the paper does a good job of synthesizing existing tools and achieves reasonably strong results with much less compute, making for a strong entry into the growing table of fast architecture search methods. I argue in favor of acceptance.
Notes:
-Whereas SMASH is limited to architectures which can be described with its proposed encoding scheme, GHNs only requires that the architecture be represented as a graph (which, to my knowledge, means it can handle any feedforward architecture). 
-Section 4.2: It's not entirely clear how this setup allows for variable sized kernels or variable #channels. Is the output of H simply as large as the largest allowable parameter tensor, and sliced as necessary? A snippet of code might be more illuminating here than a set of equations. Additionally (I may have missed this in the text) is the #channels in each node held fixed with a predfined pattern, or also searched for? Are the channels for each node within a block allowed to vary relative to one another?
-Do you sample a new, random architecture at every SGD step during training of the GHN?
-I have no expertise in graph neural networks, and I cannot judge the efficacy of this scheme wrt other GNN techniques, nor can I judge the forward-backward message passing scheme of section 4.4. If another reviewer has expertise in this area and can provide an evaluation that would be great.
 
-GPU-days is an okay metric, but it's also problematic, since it will of course depend on the choice of GPU (e.g. you can achieve a 10x speedup just from switching from a 600-series to a V100! How does using 4 GPUS for 1 hour compare to 1 GPU for 4 hours? How does this change if you have more CPU power and can load data faster? What if you're using a DL framework which is faster than your competitor's?) Given that the difference here is an order of magnitude, I don't think it matters, but if authors begin to optimize for GPU-milliseconds then it will need to be better standardized.
 
-Further empirical evidence showing the correlation between approximate performance and true performance is also strong. I very much like that this study has been run for a method based on finding paths in a larger model (ENAS) and shows that ENAS' performance does indeed correlate with true performance, *but* not perfectly, something which (if I recall correctly) is not addressed in the original paper.
 
-It is worth noting that for ImageNet-Mobile and CIFAR-10 they perform on par with the top methods but tend to use more parameters.  
-I like figures 3 and 4, the comparisons against MSDNet and random networks as a function of op budget is good to see.
-Table 4 shows that the correlation is weaker (regardless of method) for the top architectures, which I don't find surprising as I would expect the variation in performance amongst top architectures to be lower. It would be interesting to also see what the range of error rates are; I would expect that the correlation is higher when the range of error rates across the population of architectures is large, as it is easier to distinguish very bad architectures from very good architectures. Distinguishing among a set of good-to-very-good architectures is likely to be more difficult.
-For Section 5.3, I found the choice to use unseen architectures a little bit confusing. I think that even for this study, there's no reason to use a held-out set, as we seek to scrutinize the ability of the system to approximate performance even with architectures it *does* see during training. 
-How much does the accuracy drop when using GHN weights? I would like to see a plot showing true accuracy vs. accuracy with GHN weights for the random-100 networks, as using approximations like this typically results in the approximated weights being substantially worse. I am curious to see just how much of a drop there is.
-Section 5.4: it's interesting that performance is stronger when the GHN only sees a few (7) nodes during training, even though it sees 17 nodes during testing. I would expect that the best performance is attained with training-testing parity. Again, as I do not have any expertise in graph neural networks, I'm not sure if this is common (to train on smaller graphs and generalize to larger ones), so if the authors or another reviewer would like to comment and further illuminate this behavior, that would be helpful.
Some typos:
Abstract: "prematured"  should be "premature"
Introducton, last paragraph: "CNN networks." CNN already stands for Convolutional Neural Network.