Neural Architecture Optimization
Automatic neural architecture design has shown its potential in discovering powerful neural network architectures. Existing methods, no matter based on reinforcement learning or evolutionary algorithms (EA), conduct architecture search in a discrete space, which is highly inefficient. In this paper, we propose a simple and efficient method to automatic neural architecture design based on continuous optimization. We call this new approach neural architecture optimization (NAO). There are three key components in our proposed approach: (1) An encoder embeds/maps neural network architectures into a continuous space. (2) A predictor takes the continuous representation of a network as input and predicts its accuracy. (3) A decoder maps a continuous representation of a network back to its architecture. The performance predictor and the encoder enable us to perform gradient based optimization in the continuous space to find the embedding of a new architecture with potentially better accuracy. Such a better embedding is then decoded to a network by the decoder. Experiments show that the architecture discovered by our method is very competitive for image classification task on CIFAR-10 and language modeling task on PTB, outperforming or on par with the best results of previous architecture search methods with a significantly reduction of computational resources. Specifically we obtain 2.11% test set error rate for CIFAR-10 image classification task and 56.0 test set perplexity of PTB language modeling task. The best discovered architectures on both tasks are successfully transferred to other tasks such as CIFAR-100 and WikiText-2. Furthermore, combined with the recent proposed weight sharing mechanism, we discover powerful architecture on CIFAR-10 (with error rate 3.53%) and on PTB (with test set perplexity 56.6), with very limited computational resources (less than 10 GPU hours) for both tasks.
>>>>>>>>>>>>>>>>> After the discussion phase  Based on the discussion with the area chair assigned to this paper, I would like to comment on the reproducibility of the paper. In fact, I doubt that this paper is easily reproducible (which also had an influence on the score I gave to this paper). I believe (given my own experience with the reproduction of the original NAS-RL by Zoph et al and supported by Footnote 3 of the authors in the paper at hand) that too many factors on different levels are required to reproduce the results: architecture options in the target network to be optimized, hyperparameters to train the target network, architectures of the encoder, performance predictor and decoder, hyperparameters to train the encoder, performance predictor and decoder, meta-hyperparameters of Algorithm 1 and the loss function (\lambda_i). Only missing one little detail can substantially change the results (because DL is very sensitive too many hyperparameter decisions).  Because we are all humans, we tend to underestimate certain aspects and forget little, but important details to report in papers. That was again shown in this paper, since I had to ask how the authors optimized the hyperparameters of the target network. (I assume that the authors only forgot this important step and they didn’t leave it out on purpose.) Therefore, to the best of my knowledge, open source code is the best way right now to ensure reproducibility.  Another aspect is that I (as a reviewer) are not able to check whether all these important, little details are reported in the paper. If source code would be available (already during the review process), I could be sure that at least the source code covers (nearly) everything. (A docker image would be even better because the dependencies would also be listed, which can also influence the results.)  I would like to apologize in advance if I’m too sceptical here; but my experience taught me in the last years that the promise of “We will release all our codes/models after the paper is published.” is not sufficient. A discussion at the AutoML workshop@ICML’18 even supported me in my believe. Therefore, I cannot give credits to the authors because of the promise in the response letter.   >>>>>>>>>>>>>>>>> Response to response letter  Thank you for your thorough response to the reviews.  I really appreciate that you ran more experiments. Unfortunately, because of the space restrictions in the response letter, many details about the experimental setup are missing and I cannot really assess the value of these new results. Nevertheless, the results sound very promising.  Finally, I would like to disagree with some of the raised points in the response letter:  > We’d like to respectfully point out that it is neither appropriate nor necessary to directly compare NAO with ENAS.  Given that we are talking about a NIPS paper, I don't think that I'm asking for too much: a fair comparison between NAO and ENAS. I agree that both approaches are working on different aspects of the underlying problem. Nevertheless, I had the feeling that the paper has not honestly discussed how these approaches could relate and what would happen if NAO and ENAS are combined.  > Third, the EE trade-off is naturally achieved via the progress of P: initially P is inaccurate and will lead to worse architectures which plays the effect of exploration.  If that would be true, you could also argue that all the effort for exploration-exploitation trade-off in Bayesian Optimization is not necessary, because in BO, we also start with an inaccurate model. However, as shown in the BO literature, the EE-trade-off (achieved by an acquisition function) is important.   > we believe there is no need to explicitly report the result of random search  Depending on how you design your search space, random search can perform quite well. Therefore, it is an important baseline to really show that you approach can do something better.  > he performance growing w.r.t. NAO iteration (in Fig. 2(b)) cannot be achieved via random search  How do you know?  > Our method is simple grid search   I strongly wonder why people are still using grid search these days after there is so much literature on proper HPO.  > we only fine-tune  My point was that you have not reported HPO in a sufficient manner. For such an expensive approach as NAO, it is crucial how expensive HPO is, how to do HPO in this setting and last but not least, how sensitive your method is wrt to its own hyperparameters.  PS: Little details were changed in the original review.  >>>>>>>>>>>>>>>>> Original Review:  The authors address the challenge of neural architecture search. Since manual design of neural network architectures is a tedious task and requires expert knowledge, automatic neural architecture search was proposed to help new users to easily apply deep learning to their problems. The authors of the paper at hand propose an approach dubbed neural architecture *optimization* (NAO).  They call it "optimization" instead of "search" because  they optimize the architecture in a continuous space and they do not search in a discrete space. To this end, they propose to (i) learn an encoder to map architectures into a continuous space, (ii) predict the performance of architectures in the continuous space by also using a neural network and (iii) use a decoder from the continuous space into an architecture  such that they can evaluate an architecture which was predicted to have a good performance. In the end, they show that this proposed approach finds well performing architectures for CIFAR-10  and Penn Tree bank (PTB).  Most previous approaches (which are based on reinforcement learning (RL) and  evolutionary strategies (ES)) use indeed a discretized space, but the proposed new approach uses a learned continuous space to search for a good architecture. On the one hand, it is indeed an interesting paper because it proposes a completely new approach. On the other hand, the question arises whether this approach is better performing than already existing approaches or whether we can learn something interesting about the problem. Unfortunately in my opinion, the answer to both questions is no. The authors report that they found well-performing network architectures which perform similarly compared to previous approaches. However, the cost to find these networks is substantially larger than before. For example, ENAS uses 0.45 GPU days to search for a network. The proposed NAO used 800 GPU days on CIFAR-10 and 500 GPU days on PTB. So, it is not feasible in practice. Already the initialization phase of NAO is extremely expensive; the authors evaluated 600 architectures out of the 1000 networks evaluated overall only in the initial phase. Thus, the initialization phase roughly costs 480 GPU days on CIFRA-10; so roughly 1000 times the time for the entire ENAS search.  Besides the problem of the experimental results, I would like to raise several other issues:  1. The overall approaches strongly reminds me of Bayesian Optimization (BO).  The authors claim that BO is something different but they are doing something very similar:   a. based on some performance observations, they build a model   b. based on the model, they choose n architectures   c. they evaluate these architectures The main difference between BO and NAO is in fact that the authors do not use an acquisition function to trade off exploration and exploitation. However, I wonder why the authors do not have to consider this exploration-exploitation dilemma; maybe because they doing the crazy expensive initialization phase with 600 randomly sampled architectures?  2. The argument of the authors is that optimization in the discrete space is too expensive (although other researchers showed some rather convincing results) and does not scale with the number of architecture hyperparameters. However, I wonder why a mapping into a space with much more dimensions (T x d) should be more efficient. The authors do not make this point very prominent but I guess the reason could be that they can use gradient methods in the continuous space. Nevertheless, a thorough experimental proof of this point is missing. They do not even show that NAO performs better than random search with this tremendous amount of GPU days.  3. Another big issue is the handling of hyperparameter tuning, i.e., the tuning of the hyperparameter of their own NAO and all hyperparameters of the neural networks (e.g., learning rate). The authors wrote: "we have not conducted extensive hyper-parameter exploration as is typically done in previous works". So, the authors have conducted some hyperparameter optimization but they do not tell us how much exactly, how expensive it was and how the hyperparameter tuning influenced the results. Given that they changed the hyperparameters settings between CIFAR-10 and PTB (for example the trade-off parameters of their loss function), I guess that these hyperparameters are indeed crucial and NAO is not robust wrt these.  4. From the perspective of optimization (/search), the number of exemplary problems (here CIFAR-10 and PTB) is far too small. There is only little substantial evidence that the proposed approach performs well and robustly also on future problem instances. Given the advances in other communities (i.e., evaluation on tens of problems), I recommend to evaluate NAO on more datasets to obtain better empirical evidence.  I'm well aware that issues 3 and 4 also apply to many other architecture search papers. Nevertheless, these are also major flaws in these papers and it is not an sufficient argument to accept the paper at hand.  Further remarks and questions: 1. Introduction  * You can also optimize in a discrete space, e.g., the MaxSAT problem is a typical discrete optimization problem. Thus, NAS should be called NAO in the first place. And it is wrong to say that optimization in a discrete space is called "search".  * "the continuous representation of an architecture is more compact" I was missing some empirical evidence for this claim. In particular, I wonder why it should be more compact if the number of dimensions in the continuous space is larger than in the discrete space. 2. Related Work  * "Using BO, an architecture’s performance is modeled as sample from a Gaussian process (GP)" Although GPs are quite common in the BO community, approaches, such as Hyperopt and SMAC, use different models. 3. Approach  * "the outputs of all the nodes that are not used by any other nodes are concatenated to form the final output of the cell". At least in the ENAS paper, the output is not concatenated but averaged  * Why do you need all hidden states for the encoding of the continuous? Why is the last state not sufficient?  * "where X is a set containing several manually constructed architecture pairs (x 1 , x 2 )" The argument against BO was the manual design of the kernel. However, also NAO requires some manual engineering.  * Algorithm 1: How expensive is each step? Or asked differently: How large is the overhead for the embedding, the model training, the surrogate optimization and the decoding? 4. Experiments  * "For the purpose of regularization, We apply" --> we  * "The detailed results are shown in Table 4.1" -> "The detailed results are shown in Table 1" 5. Conclusion  * "makes it more effective and efficient to discover better architectures" Why is it more efficient if NAO requires much more compute resources compared to ENAS?  Last but not least, since no link to an open-source implementation is provided (or even promised) and (as the authors experienced themselves) the reproducibility in the field of NAS is rather hard, I would guess that the reproducibility of the paper at hand is also rather poor.