In this paper, we propose a deep reinforcement learning (DRL) based framework to efficiently perform runtime channel pruning on convolutional neural networks (CNNs). Our DRL-based framework aims to learn a pruning strategy to determine how many and which channels to be pruned in each convolutional layer, depending on each specific input instance at runtime. The learned policy optimizes the performance of the network by restricting the computational resource on layers under an overall computation budget. Furthermore, unlike other runtime pruning methods which require to store all channels parameters for inference, our framework can reduce parameters storage consumption for deployment by introducing a static pruning component. Comparison experimental results with existing runtime and static pruning methods on state-of-the-art CNNs demonstrate that our proposed framework is able to provide a tradeoff between dynamic flexibility and storage efficiency in runtime channel pruning.
This paper proposes to learn static and dynamic channel pruning policies for convolutional neural networks. Static pruning depends only on the training dataset and is computed once before the model is deployed. Dynamic pruning is input-dependent. The policies are obtained with deep reinforcement learning on the training dataset using a combination of the loss function and storage/computation resource budgets as a reward signal. The key novelty in this paper is to combine static and dynamic pruning which can obtain the benefits from both worlds. Experimentally, the learned pruning policies are competitive with recent dynamic pruning approaches on CIFAR-10 and ILSVRC2012, in terms of both final test accuracy and number of parameters/inference time.
Overall, I do like the idea of combining static and dynamic pruning, the DRL approach is reasonable and it seems to do well in practice. However, there are some key issues that must be addressed by the authors. In summary, these are:
1- Additional baseline methods: why not compare against a simple combination of the best published static pruning method, and one of the best published dynamic pruning methods (e.g. FBS or RNP)? I'd like to understand what the added value of jointly learning the static and dynamic pruning policies is. If a simple combination of existing static and dynamic methods works well, you will need to justify the need for the more complicated DRL approach you propose in this paper.
2- Writing: it has to be substantially improved; please see the sections Writing and Minor below.
3- Training cost: how stable is the training process described in Section 4.1? Does it much longer to train compared to pure dynamic pruning methods? 
Related work:
- Discuss connections to routing networks which adaptively route layer outputs to the next layer's modules: Rosenbaum, Clemens, Tim Klinger, and Matthew Riemer. "Routing networks: Adaptive selection of non-linear functions for multi-task learning." arXiv preprint arXiv:1711.01239 (2017).
Writing:
- Sections 3.3 up to Section 5 need lots of rewriting; I suggest some changes in "Minor" below but please make a full pass as the paper is difficult to read at the moment.
- Section 4.1 is extremely difficult to read and so I don't really understand how you train the pruning policies. Please improve the writing and summarize the process in pseudocode or illustrate it. Also, I believe 4.1 should be a section of 3 rather than in experiments. It is extremely important to understand how the network+DRL agents are trained!
Clarification:
- Number of pruned channels, runtime vs static: \ceil{d_r C} vs (C - \ceil{d_s C}); why are these different in form? Seems like the static formula prunes \ceil{(1-d_s) C}. Why is that?
- Tables 1-2: what is the "Baseline acc." and why is it different for each method? Isn't this the accuracy of the same network before any pruning?
Minor:
- Title is too long: particularly the expression "DYNAMIC FLEXIBLE RUNTIME CHANNEL". Perhaps you can think of shorter titles.
- "We consider a standard form of reinforcement learning an agent" --> "We consider a standard form of reinforcement learning: an agent"
- "are the the number of channels" --> "are the number of channels"
- "output feature F_out u_r is predicted" --> "output feature F_out, u_r, is predicted"
- "many existed dynamic" --> "many existing dynamic"
- "Since C_in is various among layers" --> "Since C_in varies among layers"
- "To avoid over-prune the filters and crashed in training, we set a minimum sparsity ratio +α, then the action space change to a_t^r ∈ (+α, 1]." --> "To avoid over-pruning the filters and crashing the training, we set a minimum sparsity ratio +α, such that the action space becomes a_t^r ∈ (+α, 1]."
- "The reward function is proposed to consider both of network accuracy and computation budget." --> "The reward function is considers both the network accuracy and computation budget."
- "and it may be various in scale" --> "and it may vary in scale"
- "large at begin of training and small near converge" --> "large at the beginning of training and small near convergence"
- "which filters can be to prune permanently" --> "which filters can be pruned permanently"
- "on two popular dataset" --> "on two popular datasets"
- "4.1 IMPLEMENT DETAILS" --> 4.1 IMPLEMENTATION DETAILS
- "For CNN fintuning" --> "For CNN finetuning"
- "Noted that for fair" --> "Note that for fair"
- Tables 1-2: "Compare to state-of-the-art" --> "Comparison to state-of-the-art"
- "Our methods has" --> "Our method has"