DARTS: DIFFERENTIABLE ARCHITECTURE SEARCH
This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Our implementation has been made publicly available to facilitate further research on efficient architecture search algorithms.
This paper proposes a novel way to formulate neural architecture search as a differentiable problem.
It uses the idea of weight sharing introduced in previous papers (convolutional neural fabrics, ENAS, and Bender et al's one shot model) and combines this with a relaxation of discrete choices between k operators into k continuous weights. Then, it uses methods based on hyperparameter gradient search methods to optimize in this space and in the end removes the relaxation by dropping weak connections and selecting the single choice of the k options with the highest weight. This leads to an efficient solution for architecture search. Overall, this is a very interesting paper that has already created quite a buzz due to the simplicity of the methods and the strong results. It is a huge plus that there is code with the paper! This will dramatically increase the paper's impact. 
In my first read through, I thought this might be a candidate for an award paper, but the more time I spent with it the more issues I found. I still think the paper should be accepted, but I do have several points of criticism / questions I detail below, and to which I would appreciate a response.
Some criticisms / questions:
1. The last step of how to move from the one-shot model to a single model is in a sense the most interesting aspect of this work, but also the one that leaves the most questions open: Why does this work? Are there cases where we lose arbitrarily badly by rounding the solution to the closest discrete value or is the performance loss bounded? How would other ways of moving from the relaxation to a discrete choice work? I don't expect the paper to answer all of these questions, but it would be useful if the authors acknowledge that this is a critical part of the work that deserves further study. Any insights from other approaches the authors may have tried before the mechanism in Section 2.4 would also be useful.
2. The related work is missing several papers, namely the entire category of work on using network morphisms to speed up the optimization process, Bender et al's one shot model, and several early papers on neural architecture search (work on NAS did not only start in 2017 but goes back to work in the 1990s on neuroevolution that is very similar to the evolution approach by Real). This is a useful survey useful for further references: https://arxiv.org/abs/1808.05377
3. I find a few of the claims to be a bit too strong. In the introduction, the paper claims to outperform ENAS, but really the paper doesn't give a head-to-head comparison. In the experiments, ENAS is faster and gives slightly worse results. The authors state explicitly that their method is slower because they run it 4 times and pick the best result. One could obviously also do that with ENAS, and since ENAS is 8 times faster one could even run it 8 times! This is unfair and should be fixed. I don't really care even if it turns out that ENAS performs a bit better with the same budget, but comparisons should be fair and on even ground in order to help our science advance -- something that is far too often ignored in the ML literature in order to obtain a table with bold numbers in one's own row.
Likewise, why is ENAS missing in the Figure 3 plots for CIFAR, and why is its performance not plotted over time like that of DARTS?
4. The paper is not really forthcoming about clearly stating the time required to obtain the results:
- On CIFAR, there are 4 DARTS run of 1 day each
- Then, the result of each of these is evaluated for 100 epochs (which is only stated in the caption of Figure 3) to pick the best. Each of these validation runs takes 4 hours (which, again, one has to be inferred from the fact that random search can do 24 such evaluations in 4 GPU days), so this step takes another 16 GPU hours.
- Then, one needs to train the final network for 600 epochs; this is a larger network, so this should take another 2-3 GPU days.
So, overall, to obtain the result on CIFAR-10 requires about one GPU week. That's still cheap, but it's a different story than 1 day.
Likewise, DARTS is *not* able to obtain 55.7 perplexity on PTB in 6 hours with 4 GPUs; again, there is the selection step (probably another 4*6 hours?) and I think training the final model takes about 2 GPU days. These numbers should be stated prominently next to the stated "search times" to not mislead the reader.
5. One big question I have is where the hyperparameters come from, for both the training pipeline and the final evaluation pipeline (which actually differ a lot!).
For example, here are the hyperparameters for CIFAR, in this format: training pipeline value -> final evaluation pipeline value:
#cells: 8 -> 20
batch size: 64 -> 96
initial channels: 16 -> 36
#epochs: 50 -> 600
droppath: no -> yes (with probability 0.2)
auxiliary head: no -> yes (with weight 0.4)
BatchNorm: enabled (no learnable parameters) -> enabled
The situation is similar for PTB:
embedding size: 300 -> 850
hidden units per RNN layer: 300 -> 850
#epochs: 500 -> 8000
batch size: 256 (SGD) -> 64 (ASGD), sped up by starting with SGD
weight decay: 5e-7 -> 8e-7
BatchNorm: enabled (no learnable parameters) -> disabled
The fact that there are so many differences in the pipelines is disconcerting, since it looks like a lot of manual work is required to get these right. Now you need to tune hyperparameters for both the training and the final evaluation pipeline? If you have to tune them for the final evaluation pipeline, then you can't capitalize at all on the fact that DARTS is fast, since hyperparameter optimization on the full final evaluation pipeline will be order of magnitudes more expensive than running DARTS.
6. How was the final evaluation pipeline chosen? Before running DARTS the first time, or was it chosen to be tuned for architectures found by DARTS?
7. A question about how the best of 4 DARTS runs is selected, and how the best of the 24 random samples in random search is evaluated: is this based on 100 epochs using the *training* procedure or the *final evaluation* procedure? Seeing how different the hyperparameters are above, this should be stated.
8. A few questions to the authors related to the above: how did you choose the hyperparameters of DARTS? The DARTS learning rate for PTB is 10 times higher than for CIFAR-10, and the momentum also differs a lot (0.9 vs. 0.5). Did you ever consider different hyperparameters for DARTS? If so, how did you decide on the ones used? Is it sensitive to the choice of hyperparameters? In the author response period, could you please report the  
(1) result of running DARTS on PTB using the same DARTS hyperparameters as used for CIFAR-10 (learning rate 3*e-4 and momentum (0.5,0.999)) and
(2) result of running DARTS on CIFAR-10 using the same DARTS hyperparameters as used for PTB (learning rate 3*e-3 and momentum (0.9,0.999))?
9. DARTS is being critizized in https://openreview.net/pdf?id=rylqooRqK7#page=10&zoom=180,-16,84
I am wondering whether the authors have a reply to this.
The algorithm for solving the relaxed problem is also not mathematically derived from the optimization problem to be solved (equations 3,4), but it is more a heuristic. A derivation, or at least a clearer motivation for the algorithm would be useful.
10. Further comments:
- Equation 1: This looks like a typo, shouldn't this be x(j) = \sum_{i<j} o(i,j) x(i) ? Even if the authors wanted to use the non-intuitive way of edges going from j to i, then o(i,j) should still be o(j,i).
- Just above Equation 5: "the the"
- Equation 5: I would have found it more intuitive had \alpha_{k-1} already just been a generic \alpha here.
- It would be nice if the authors gave the explicit equations for the extension with momentum in the appendix for completeness.
- The authors should include citations for techniques such as batch normalization, Adam, and cosine annealing.
Despite these issues (which I hope the authors will address in the author response and the final version), as stated above, I'm arguing for accepting the paper, due to the simplicity of the method combined with its very promising results and the direct availability of code.