Published as a conference paper at ICLR 2018 N2N LEARNING: NETWORK TO NETWORK COMPRESSION VIA POLICY GRADIENT REINFORCEMENT LEARNING
While wider and deeper neural network architectures continue to advance the state-of-the-art for many computer vision tasks, real-world adoption of these networks is impeded by hardware and speed constraints. Conventional model compression methods attempt to address this problem by modifying the architecture manually or using pre-defined heuristics. Since the space of all reduced architectures is very large, modifying the architecture of a deep neural network in this way is a difficult task. In this paper, we tackle this issue by introducing a principled method for learning reduced network architectures in a data-driven way using reinforcement learning. Our approach takes a larger 'teacher' network as input and outputs a compressed 'student' network derived from the 'teacher' network. In the first stage of our method, a recurrent policy network aggressively removes layers from the large 'teacher' model. In the second stage, another recurrent policy network carefully reduces the size of each remaining layer. The resulting network is then evaluated to obtain a reward -a score based on the accuracy and compression of the network. Our approach uses this reward signal with policy gradients to train the policies to find a locally optimal student network. Our experiments show that we can achieve compression rates of more than 10× for models such as ResNet-34 while maintaining similar performance to the input 'teacher' network. We also present a valuable transfer learning result which shows that policies which are pre-trained on smaller 'teacher' networks can be used to rapidly speed up training on larger 'teacher' networks.
Summary:
The manuscript introduces a principled way of network to network compression, which uses policy gradients for optimizing two policies which compress a strong teacher into a strong but smaller student model. The first policy, specialized on architecture selection, iteratively removes layers, starting with architecture of the teacher model. After the first policy is finished, the second policy reduces the size of each layer by iteratively outputting shrinkage ratios for hyperparameters such as kernel size or padding. This organization of the action space, together with a smart reward design achieves impressive compression results, given that this approach automates tedious architecture selection. The reward design favors low compression/high accuracy over high compression/low performance while the reward still monotonically increases with both compression and accuracy. As a bonus, the authors also demonstrate how to include hard constraints such as parameter count limitations into the reward model and show that policies trained on small teachers generalize to larger teacher models.
Review:
The manuscript describes the proposed algorithm in great detail and the description is easy to follow. The experimental analysis of the approach is very convincing and confirms the author’s claims. 
Using the teacher network as starting point for the architecture search is a good choice, as initialization strategies are a critical component in knowledge distillation. I am looking forward to seeing work on the research goals outlined in the Future Directions section.
A few questions/comments:
1) I understand that L_{1,2} in Algorithm 1 correspond to the number of layers in the network, but what do N_{1,2} correspond to? Are these multiple rollouts of the policies? If so, shouldn’t the parameter update theta_{{shrink,remove},i} be outside the loop over N and apply the average over rollouts according to Equation (2)? I think I might have missed something here.
2) Minor: some of the citations are a bit awkward, e.g. on page 7: “algorithm from Williams Williams (1992). I would use the \citet command from natbib for such citations and \citep for parenthesized citations, e.g. “... incorporate dark knowledge (Hinton et al., 2015)” or “The MNIST (LeCun et al., 1998) dataset...” 
3) In Section 4.6 (the transfer learning experiment), it would be interesting to compare the performance measures for different numbers of policy update iterations.
4) Appendix: Section 8 states “Below are the results”, but the figure landed on the next page. I would either try to force the figures to be output at that position (not in or after Section 9) or write "Figures X-Y show the results". Also in Section 11, Figure 13 should be referenced with the \ref command
5) Just to get a rough idea of training time: Could you share how long some of the experiments took with the setup you described (using 4 TitanX GPUs)?
6) Did you use data augmentation for both teacher and student models in the CIFAR10/100 and Caltech256 experiments?
7) What is the threshold you used to decide if the size of the FC layer input yields a degenerate solution?
Overall, this manuscript is a submission of exceptional quality and if minor details of the experimental setup are added to the manuscript, I would consider giving it the full score.