In many learning situations, resources at inference time are much more constrained than resources at training time. This paper studies a general paradigm, called Differentiable ARchitecture Compression (DARC), that combines model compression and architecture search to learn models that are resource-efficient at inference time. Given a resource-intensive base architecture, DARC utilizes the training data to learn which sub-components can be replaced by cheaper alternatives. The high-level technique can be applied to any neural architecture, and we report experiments on state-of-the-art convolutional neural networks for image classification. For a WideResNet with 97.2% accuracy on CIFAR-10, we improve single-sample inference speed by 2.28× and memory footprint by 5.64×, with no accuracy loss. For a ResNet with 79.15% Top1 accuracy on ImageNet, we improve batch inference speed by 1.29× and memory footprint by 3.57× with 1% accuracy loss. We also give theoretical Rademacher complexity bounds in simplified cases, showing how DARC avoids overfitting despite over-parameterization.
This paper addresses the problem of limited resources that may generally be available at inference time (as compared to the lack of constraints during training time). The paper addresses architecture search as well as model compression by simultaneously optimizing all layers/weights/submodules of the network. The approach is motivated well in terms of comparisons to other work: Although I am not an expert in this particular area of model compression, the intuitive comparisons and mentions of prior work felt satisfying and informative. This problem is definitely timely, and the paper showed results of improved inference speed and memory footprint on both a smaller 10-way classification task as well as a larger 1000-way one. 
My current decision is a weak accept, for a well-written paper and an interesting problem, but in the presence of some concerns and suggestions as listed below. 
(1) My main concern is that without weight regularization, this optimization problem seems ill-posed. Looking at algorithm 1 specifically, consider the case where j=10 and the learned alphas are indeed sparse, like (0.1, 0, 0,…, 0, 0.9). The cost of J=0 here might be very high (i.e., large, slow, expensive model) and cost of J=9 might be very low (i.e., small fully-connected network that basically maps to an identity function). Here, the cost during optimization could be very low, and the results could be very accurate, but the weights of J=0 may be very high such that the entire work of getting the right answer is from this model. More generally, a low alpha can be attained for any model by simply increasing the network weights themselves. This doesn’t seem to be taken care of anywhere in this paper; if not addressed in any way, it would be a large reason for me to suggest rejecting this work.
(2) For the experiments, both of the chosen tasks were classification. Although one was smaller/easier and one was larger/harder, it would serve to be much more convincing if a task requiring regression and/or higher-dimensional output were tested. For these classification tasks, I can imagine that for certain model changes, the decision boundaries may remain the same even if the network itself is changing in detrimental ways. Thus, a task such as an image-to-image depth perception task (or other tasks in this category) would solidify the results and make it more convincing that DARC does indeed help.
Minor: 
(a) The motivation of going from L0 to L1 for the cost constraint seems relevant also to the choice of h being the convex combination of different models. Can the motivation and problem setup address these motivations together, instead of separately?  
(b) Although this paper defined R(h) and it also defined h=(\sum (alpha_j * h_j), it would still be helpful for the reader if the paper could explicitly define R(\sum (alpha_j * h_j)) somewhere. Although I know what was intended was to compose these h_j functions as sequential stages to pass data through, the sum notation seems a bit misleading like you may be averaging the final predictions/outputs from different models and computing the loss on these averaged results.