Recent years have witnessed growing interests in designing efficient neural networks and neural architecture search (NAS). Although remarkable efficiency and accuracy have been achieved, existing expert designed and NAS models neglect the fact that input instances are of varying complexity and thus different amounts of computation are required. Inference with a fixed model that processes all instances through the same transformations would incur computational resources unnecessarily. Customizing the model capacity in an instance-aware manner is required to alleviate such a problem. In this paper, we propose a novel Instanceaware Selective Branching Network-ISBNet to support efficient instance-level inference by selectively bypassing transformation branches of insignificant importance weight. These weights are dynamically determined by a lightweight hypernetwork SelectionNet and recalibrated by gumbel-softmax for sparse branch selection. Extensive experiments show that ISBNet achieves extremely efficient inference in terms of parameter size and FLOPs comparing to existing networks. For example, ISBNet takes only 8.70% parameters and 31.01% FLOPs of the efficient network MobileNetV2 with comparable accuracy on CIFAR-10.
This paper proposes an instance-aware dynamic network, ISBNet, for efficient image classification. The network consists of layers of cell structures with multiple branches within. During the inference, the network uses SelectionNet to compute a "calibration weight matrix", which essentially controls which branches within the cell should be used to compute the output. Similar to previous works in NAS, this paper uses Gumbel Softmax to compute the branch selection probability. The network is trained to minimize a loss function that considers both the accuracy and the inference cost. Training of the network is divided into two stages: First, a high temperature is used to ensure all the branches are sufficiently optimized, and at the second stage, the authors aneal the temperature. During the inference, branches are selected if their probability computed by Gumbel Softmax is larger than a certain threshold.
Overall, the idea of the paper is clearly presented. The methods used in this paper are similar to previous works on neural architecture search (NAS), but this paper can be seen as a meaningful extension to NAS. 
My main concern for this paper is the experiment section.
1) The paper claims that "ISBNet takes only 8.70% parameters and 31.01% FLOPs of the efficient network MobileNetV2 with comparable accuracy on CIFAR-10". Mainstream efficient networks, such as MobileNet and ShuffleNet, are not designed for CIFAR-10 datasets, but ImageNet datasets. Their downsampling strategy is very aggressive, which leads to relatively poor accuracy on CIFAR-10 datasets with 32x32 images. Therefore, it is not fair to compare the MobileNetV2 on CIFAR-10 and claim superiority over it. Compared with other networks customized for the CIFAR-10 dataset, such as NASNet-A, DARTS, and so on, the error rate of ISBNet is significantly worse (>1% higher than DARTS, or >33% relative increase in error rate). 
2) In table 2, the paper compares ISBNet's performance on the ImageNet dataset with other baselines. However, the baseline models are not up to date. For example, MobileNetV2 achieves a 28% error rate with 300M FLOPs, FBNet achieves a 27% error rate with 249M FLOPs. These results, however, are not cited in this paper. Particularly, MobileNetV2 is compared against on the CIFAR-10 dataset, but not the ImageNet dataset. 
3) ISBNet is not the first instance-aware dynamic network. Previous works such as SkipNet, Soft-Conditional computing [1] explored similar ideas. However, in this paper, there is no comparison with previous dynamic networks. 
Overall, I would expect a much stronger experiment section for the paper to be published. 
[1] https://arxiv.org/abs/1904.04971