Neural Architecture Search (NAS) has shown great potentials in finding a better neural network design than human design. Sample-based NAS is the most fundamental method aiming at exploring the search space and evaluating the most promising architecture. However, few works have focused on improving the sampling efficiency for a multi-objective NAS. Inspired by the nature of the graph structure of a neural network, we propose BOGCN-NAS, a NAS algorithm using Bayesian Optimization with Graph Convolutional Network (GCN) predictor. Specifically, we apply GCN as a surrogate model to adaptively discover and incorporate nodes structure to approximate the performance of the architecture. For NAS-oriented tasks, we also design a weighted loss focusing on architectures with high performance. Our method further considers an efficient multi-objective search which can be flexibly injected into any sample-based NAS pipelines to efficiently find the best speed/accuracy trade-off. Extensive experiments are conducted to verify the effectiveness of our method over many competing methods, e.g. 128.4× more efficient than Random Search and 7.8× more efficient than previous SOTA LaNAS for finding the best architecture on the largest NAS dataset NASBench-101.
This paper proposed BOGCN-NAS that encodes current architecture with Graph convolutional network (GCN) and uses the feature extracted from GCN as the input to perform a Bayesian regression (predicting bias and variance, See Eqn. 5-6). They use Bayesian Optimization to pick the most promising next model with Expected Improvement, train it and take its resulting accuracy/latency as an additional training sample, and repeat. 
They tested the framework on both single-objective and multi-objective tasks.  On the single-objective (accuracy task). They tested it on NasBench and LSTM-12K, two NAS datasets with pre-trained models and their performance. They obtained very good performance on both, beating LaNAS (previous SoTA) by 7.8x higher sample efficiency. On multiple-objective, they show higher efficiency in finding Pareto frontier models, compared to random search. 
One main question I have is whether the next model is chosen given the current prediction model? For NasBench, did you run your predictor for all (420k minus explored) models and pick the one that maximizes Expected Improvement? Note that LaNAS is more efficient in that manner by sampling directly on polytopes formed by linear constraints. If so, how do you pick the next candidate models in open domain setting?
It looks like Eqn. 9 biases the training heavily towards high accuracy region, which is a hack. Although in the Appendix (Fig. 7) the authors have already perform some analysis on the effect of different weight terms, I wonder whether there is a more problem-independent way of doing it. The MCTS in LaNAS is one way that automatically focuses the search on the important regions. Currently, the proposed approach might limit the usability of the proposed method to other situations when accuracy is no longer that important. 
The performance is really impressive in the NasBench and LSTM dataset. The paper mentioned that “BO can really predict the precious model to explore next” but didn’t provide an examples in the paper. I would whether the author could visualize it in the next revision, which would be very cool. 
Do you have open domain search research in single-objective search? 
Why not use NasNet architecture for a fair comparison with other NAS papers?
In Appendix, Fig. 6 shows that even without GCN and BO, a single MLP already achieves global optimal solution in LSTM-12K dataset with ~850 samples, already beating all the previous methods (Random, Reg Evolution, MCTS and LaNAS). If that’s the case, I wonder how much roles the proposed methods (BO + GCN) play during search? Also what do you mean by “without BO”? Do you only predict the mean and assume all variance is constant? 
=====Post Rebuttal======
I have read other reviewers' comments and the rebuttal. 
One of the main problems in this paper is an unfair comparison against LaNAS. LaNAS only uses a single sample at each leaf, while they sample 100% to 0.1% of the models, evaluate them with the current BO model and find the best. For NasBench-101 with 420K models, even sampling 0.1% each time means ~400 samples and the performance (from the rebuttal) seems to degrade substantially from 100% case (1464.4->4004.4). This means that almost 3x more samples are needed, compared to what they claimed. 
I agree with the authors that calling BO function is super fast so maybe this is fine. However, on the open domain experiments, their performance is also not better than LaNAS+c/o, which they didn't list in the rebuttal. I listed it here:
Model                      	Params       	Top-1 err	No. of samples truly evaluated
-----------------------------------------------------------------------------------------------------------------
BOGCN+cutout (V1)	   3.1M 	          2.74	                      200
BOGCN+cutout (V2)	   3.5M 	          2.61	                      400
LaNAS+c/o                       3.2M              2.53±0.05                    803
Overall this paper is on the borderline. I don't mind if the paper gets rejected. For now I lower the score to 3.