Runtime Neural Pruning
In this paper, we propose a Runtime Neural Pruning (RNP) framework which prunes the deep neural network dynamically at the runtime. Unlike existing neural pruning methods which produce a fixed pruned model for deployment, our method preserves the full ability of the original network and conducts pruning according to the input image and current feature maps adaptively. The pruning is performed in a bottom-up, layer-by-layer manner, which we model as a Markov decision process and use reinforcement learning for training. The agent judges the importance of each convolutional kernel and conducts channel-wise pruning conditioned on different samples, where the network is pruned more when the image is easier for the task. Since the ability of network is fully preserved, the balance point is easily adjustable according to the available resources. Our method can be applied to off-the-shelf network structures and reach a better tradeoff between speed and accuracy, especially with a large pruning rate.
Update: I updated my score after RNN overhead clarification for more recent networks and their remark on training complexity.
The authors propose a deep RL based method to choose a subset of convolutional kernels in runtime leading to faster evaluation speed for CNNs. I really like the idea of combining an RNN and using it to guide network structure. I have some doubts on the overhead of the decision network (see below) would like a comment on that before making my final decision.
-Do your computation results include RNN runtime (decision network), can you comment on the overhead? 
The experiments focus on VGG16 which is a very heavy network. In comparison, more efficient networks like GoogleNet may make the RNN overhead significant. Do you have time or flops measurements for system overhead? Finally, as depth increases (e.g. Resnet151) your decision overhead increases as well since the decider network runs at every step.
- Do you train a different RNP for each p? Is 0.1 a magic number?
-The reward structure and formulating feature (in this case kernel) selection as an MDP is not new, check Timely Object Recognition Karayev, Imitation Learning by Coaching, Hal Daume III. However, application of it to internal kernel selection is interesting.
line 33 "Some other samples are more difficult, which require more computational resources. This property is not exploited in previous neural pruning methods, where input samples are treated equally" This is not exactly true check missing references below.
Missing references on sample adaptive inference in DNNs:
-The cascading neural network: building the Internet of Smart Things Leroux et al.
-Adaptive Neural Networks for Fast Test-Time Prediction Bolukbasi et al.
-Changing Model Behavior at Test-Time Using Reinforcement Learning Odena et al.
misc:
table 1 3x column should have Jaderberg as bold (2.3 vs 2.32)
line 158 feature -> future
line 198 p=-0.1 -> p=0.1 ?