In this paper, we propose an efficient framework to accelerate convolutional neural networks. We utilize two types of acceleration methods: pruning and hints. Pruning can reduce model size by removing channels of layers. Hints can improve the performance of student model by transferring knowledge from teacher model. We demonstrate that pruning and hints are complementary to each other. On one hand, hints can benefit pruning by maintaining similar feature representations. On the other hand, the model pruned from teacher networks is a good initialization for student model, which increases the transferability between two networks. Our approach performs pruning stage and hints stage iteratively to further improve the performance. Furthermore, we propose an algorithm to reconstruct the parameters of hints layer and make the pruned model more suitable for hints. Experiments were conducted on various tasks including classification and pose estimation. Results on CIFAR-10, ImageNet and COCO demonstrate the generalization and superiority of our framework.
This paper proposes a new framework which combines pruning and model distillation techniques for model acceleration. Though the ``pruning” (Molchanov et al. (2017)) and hint components already exists, the authors claim to be the first to combine them, and experimentally show the benefit of jointly and iteratively applying the two techniques. The authors show better performance of their new framework over baseline, pruning only method and hint only method on a few standard Vision data set.
The motivation is clearly conveyed in the paper. As a framework of combining two existing techniques, I expect the framework can stably improve its two components without too much additional time cost. I have some small questions.
--What is the ``additional cost” of your proposed framework. For example, how many iterations do you typically use. For each data set, what time delta you spent to get the performance improvement comparing to pruning only or hint only models.
--In your iterative algorithm (pseudo code in appendix), the teacher model is only used in the very beginning and final step, though richest information is hidden in the original teacher model. In the intermediate steps, you are fine tuning iteratively without accessing the original teacher model.
--In your reconstruction step, you said due to the randomness, you do not always use the learned new W. How much your algorithm benefit from this selection strategy?