The architecture search methods for convolutional neural networks (CNNs) have shown promising results. These methods require significant computational resources, as they repeat the neural network training many times to evaluate and search the architectures. Developing the computationally efficient architecture search method is an important research topic. In this paper, we assume that the structure parameters of CNNs are categorical variables, such as types and connectivities of layers, and they are regarded as the learnable parameters. Introducing the multivariate categorical distribution as the underlying distribution for the structure parameters, we formulate a differentiable loss for the training task, where the training of the weights and the optimization of the parameters of the distribution for the structure parameters are coupled. They are trained using the stochastic gradient descent, leading to the optimization of the structure parameters within a single training. We apply the proposed method to search the architecture for two computer vision tasks: image classification and inpainting. The experimental results show that the proposed architecture search method is fast and can achieve comparable performance to the existing methods.
The authors propose to formulate the neural network architecture as a collection of multivariate categorical distributions. They further derive sample-based gradient estimators for both the stochastic architecture and the deterministic parameters, which leads to a simple alternating algorithm for architecture search.
Pros:
+ Intuitions and formulations are easy to comprehend.
+ Simpler to implement than most prior methods.
+ Appealing results (on CIFAR-10) as compared to the state-of-the-art.
Cons:
- Limited technical novelty. The approach is a straightforward extension of Shirakawa et al. 2018. The main algorithm is essentially the same except minor differences in gradient derivations.
- Lack of theoretical justifications. It seems all the derivations at the beginning of Section 2 assume the architecture is optimized wrt the training set. However, the authors ended up splitting the dataset into two parts in the experiments and optimize the architecture wrt a separate validation set instead. This would invalidate all the previous derivations.
- The method is a degenerated version of ENAS. A closer look at eq (2) and (3) suggests the resulting iterative algorithm is almost the same as that in ENAS, where the weights are optimized using GD wrt the training set and the architecture is optimized using the log-derivative trick wrt the validation set. The only distinction are (i) using a degenerated controller/policy formulated as categorical distributions (ii) using the validation loss instead of the validation accuracy as the reward (according to eq. (3)). This is also empirically reflected in Table 1, which shows the proposed PDAS is similar to ENAS both in terms of efficiency and performance. The mathematical resemblance with ENAS is not necessarily bad, but the authors need to make it more explicit in the paper. 
Minor issues:
* I'm not sure whether it's a good practice to report the "best" test error among multiple runs in Table 1.
* The method is not really "parameterless" as claimed in the introduction. For example, a suitable learning rate adaptation rule can be task-specific thus requires manual tuning/design. The method also consists of some additional hyperparameters like the \lambda in the utility transform.