Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-net and searching for an optimal architecture. In this paper, we present a novel approach, namely Partially-Connected DARTS, by sampling a small part of super-net to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance. In particular, we perform operation search in a subset of channels while bypassing the held out part in a shortcut. This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels. We solve it by introducing edge normalization, which adds a new set of edge-level hyper-parameters to reduce uncertainty in search. Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoy both faster speed and higher training stability. Experiment results demonstrate the effectiveness of the proposed method. Specifically, we achieve an error rate of 2.57% on CIFAR10 within merely 0.1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.2% on ImageNet (under the mobile setting) within 3.8 GPU-days for search. Our code has been made available at https://www.dropbox.com/ sh/on9lg3rpx1r6dkf/AABG5mt0sMHjnEJyoRnLEYW4a?dl=0.
---
revised score. rebuttal clears my concerns.
---
Summary:
The paper proposes a partially connected differential architecture search (PC-DARTS) technique, that uses a variant of channel dropout for each node's output feature maps, and a weighted summation of concatenating all previous nodes. Searched architecture on CIFAR-10 and ImageNet seems to outperform the one discovered by the original DARTS, however, the results are not directly comparable due to the slight change of search space. 
Introducing this edge normalization is a novel contribution, but it is more like a trick to have a better search space rather than the PC-DARTS itself. My main concerns are about the incremental novelty and experiments are heavily done on one search run, especially the search space is not the same as baseline DARTS. 
I do not think the current version is ready for ICLR, but I am looking forward to seeing the authors' rebuttal and I am willing to revise my review accordingly.
Main concerns
- Incremental novelty about channel sampling.
Doing edge normalization in the PC-DARTS is indeed novel, however, the channel sampling (abbr. PC for partial channel connection) is not. Dropout is widely adopted in all deep learning training since AlexNet. In NAS with parameter sharing, Pham et al. already exploit the channel dropout as shown in ENAS function "def drop_path"(https://github.com/melodyguan/enas/blob/master/src/cifar10/image_ops.py). It is true that previous works treated like one hyper-parameters and do not provide deeper insight about this term, but it is not correct to say in Section 3.4 "Channel sampling ... has never been studied in prior work". In my perspective, the key difference of channel sampling is the retained channel number is always fixed to K, the non-selected channels are not zeroed, where the dropout usually has only a probability K / total_channel and non-selected feature is multiplied to a zero constant. 
Thus, I suggest authors provide additional experiments as in Table 3 to compare the original drop-path with proposed channel sampling. Considering the test error drops from 3% to 2.67% while using PC, it will be more convincing to show the original drop path with probability K / total_channel yields a smaller drop to evidence the effectiveness of proposed sampling. 
- Proposed edge normalization is not a new sampling policy but a new search space.
To my understanding, this edge normalization is effectively a change to the search space rather than the sampling policy, and generalize to many other policies as well, and can be a substantial contribution to the NAS community. However, under current experiments setting, it is hard to isolate the improvement is from this new space or the channel sampling, as detailed later.
- About the motivation.
Throughout the paper, in abstract, introduction, section 3.2 and section 4.4, the authors claim that the larger batch size is particularly important for the stability of architecture search which is not well-studied and lack of references. From Table 4, it is hard to tell the stability is from the larger batch size or the proposed partial channel sampling.
- Questions about experiments
1. Experiments comparing to the baseline is not fair. 
As in Section 4.2, the CIFAR-10 search is different from the original DARTS and P-DARTS in the following manner. The batch size is changed from 64(in DARTS)/96(in P-DARTS) to 256,  super-net is freezed for the first 15 epochs, and introducing the edge normalization parameter \beta_{i,j} increase the search space.  With all these changes, it is quite hard to isolate the effectiveness of proposed PC-DARTS. Two possible simple experiments to compare is, using the original DARTS space and training set, 1) do not update the \beta but use a fixed initialization that all \beta is the same (to mimic original DARTS concatenation); 2) add \beta to original DARTS as well and re-run 1). 
It is completely reasonable to me the contribution of this paper is introducing a novel edge-normalization that is simple and effective to improve the DARTS based approach. If so, the authors could revise the conclusion easily. However, in the least scenario, the experiment comparison should be in a fair way.
2. In original DARTS, error drop from 3% for the first-order gradient to 2.76% while using the second-order one, will this trend occurs with PC-DARTS?
3. Robustness 
Recent work about evaluating neural architecture search reveals that NAS algorithms are sensitive to random initialization[1,2] and the search space [3], this in general leads to a notorious reproducibility problem of current NAS and shows it is not reasonable to only compare final performances on proxy tasks over **one** searched architecture. However, in the stability study in Section 4.4.3, multiple runs are still over the same architecture discovered in earlier experiments. In Section 4.4.2, the paper mentioned the search runs multiple times, yet the reported results in Table 3 are against the single run, as indicated by CIFAR-10 no PC- no EN error 3.00 +- 0.14, which is identical to the results in Table 1 DARTS (1st-order). Could the authors report the results with at least 3 different initializations, and possibly release the seeds? It would significantly strengthen the effectiveness of the proposed approach.
Minor comments
- According to Section 4.4.1 and Figure 3, change the K from 1 to 8, the search cost drops significantly. Does this mean the batch size in the ablation study is changing all the time? How could we know if the test-error is reduced due to the sampling ratio or to the batch size? 
Typos
1. Table 2, caption below the table, \dag is not aligned with the one in the used table.
--- reference ---
[1] Li and Talwalker, Random search and reproducibility of neural architecture search, UAI’19
[2] Sciuto et al., Evaluating the search phase of neural architecture search, arxiv’19
[3] Radosavovic et al., On Network Design Spaces for Visual Recognition, ICCV'19.