Bottleneck structures with identity (e.g., residual) connection are now emerging popular paradigms for designing deep convolutional neural networks (CNN), for processing large-scale features efficiently. In this paper, we focus on the information-preserving nature of identity connection and utilize this to enable a convolutional layer to have a new functionality of channel-selectivity, i.e., redistributing its computations to important channels. In particular, we propose Selective Convolutional Unit (SCU), a widely-applicable architectural unit that improves parameter efficiency of various modern CNNs with bottlenecks. During training, SCU gradually learns the channel-selectivity on-the-fly via the alternative usage of (a) pruning unimportant channels, and (b) rewiring the pruned parameters to important channels. The rewired parameters emphasize the target channel in a way that selectively enlarges the convolutional kernels corresponding to it. Our experimental results demonstrate that the SCU-based models without any postprocessing generally achieve both model compression and accuracy improvement compared to the baselines, consistently for all tested architectures.
This is an architecture design paper. It proposes a general structure called Selective Convolutional Unit that the authors claim to be useful for various CNN models. The SCU structure contains two major parts: CD and NC. CD for compressing/pruning channels and NC for multiplicative noise. The paper gives a measure, called expected channel damage score, on the change of the output for SCU. It also shows the effectiveness of SCU on CIFAR-10, CIFAR-100 and imagenet.
Some questions and concerns:
1. The paper spends too much space introducing the bottleneck structures and a whole lot of the details on the optimization of NC and CD are put in the appendix. I would suggest to reduce the section of introductory part and put a shorter version of appendix A and B to the main text so that the readers know more about the architecture and how it is optimized. In particular, the description on NC is confusing since without looking at the appendix it is not clear how the prior p(\theta) is used. 
2. The experiment shows improvement on densenet and resnetXT, but the result is not the state-of-the-art. Wide-Resnet seems to get better accuracy on both CIFAR-10 and CIFAR-100 compared to the best accuracy reported in this paper. Also the number reported by the original densenet paper on imagenet seems to be better (densenet-264 has an error rate of 22.15/20.80)
3. In your CD design, channel assignment \pi is a discrete variable. How is it optimized in the training process?
4. The proof of proposition 1 does not look correct to me. The optimization procedure makes use of the data X to determine your NC variable \theta so \theta depends on X. In this way you cannot factorize the expectation in the equation below (20) in your appendix.