Moonshine: Distilling with Cheap Convolutions
Many engineers wish to deploy modern neural networks in memory-limited settings; but the development of flexible methods for reducing memory use is in its infancy, and there is little knowledge of the resulting cost-benefit. We propose structural model distillation for memory reduction using a strategy that produces a student architecture that is a simple transformation of the teacher architecture: no redesign is needed, and the same hyperparameters can be used. Using attention transfer, we provide Pareto curves/tables for distillation of residual networks with four benchmark datasets, indicating the memory versus accuracy payoff. We show that substantial memory savings are possible with very little loss of accuracy, and confirm that distillation provides student network performance that is better than training that student architecture directly on data.
Main idea: this paper proposes to tackle the task of speeding up any CNN by replacing the "higher level blocks" that typically consist of stacked convolutions with cheaper blocks consisting of 1x1 convolutions and potentially group convolutions. Thus far this is not super-novel, as this idea has been studied before. However, the authors also employ knowledge distillation (again this idea has been studied before in isolation) to match the inner layer block activations between the teacher and the student (cheaper) networks. This shows very promising results over the baseline which only replaces the blocks with their cheaper version without distillation.   The main selling point of this paper is that the ideas have been proven before independently, and this paper ties them together nicely. Therefore, it should be relatively easy for other researchers to hopefully replicate these results. Another nice plus is that the authors included a segmentation task besides classification. This is important because in the literature most of the work done in speeding up networks seems to be centered around classification, and it's not clear whether such methods generalize beyond this niche domain. In fact, from personal experience, we have also experienced a huge quality degradation when changing the convolutions to be depthwise, but not doing distillation in the context of non-classification problems.  On the negative side, I would have wished to see a discussion about runtime. Most of the focus of the paper is in comparing the number of parameters against the various errors. For this to be truly useful in practice it would be great to discuss how does the number of FLOPs (or forward pass time) change (did it increase?, did it decrease?). I would strongly suggest the authors clarify this in the final version of the paper.  Overall score: I think the paper has some good practical applications, and I think we should accept it.  Confidence: I am not super familiar with the field of making networks faster besides the ideas I mentioned above, and I did do a bit of research on Google Scholar trying to find similar papers that discuss both ideas, but I didn't find anything matching quite exactly (e.g., MobileNets seem to use both distillation and cheaper convolutions but they don't match the internal features).