The availability of general-purpose reference and benchmark datasets such as ImageNet have spurred the development of general-purpose popular reference model architectures and pre-trained weights. However, in practice, neural networks are often employed to perform specific, more restrictive tasks, that are narrower in scope and complexity. Thus, simply fine-tuning or transfer learning from a generalpurpose network inherits a large computational cost that may not be necessary for a given task. In this work, we investigate the potential for model specialization, or reducing a model's computational footprint by leveraging task-specific knowledge, such as a restricted inference distribution. We study three methods for model specialization-1) task-aware distillation, 2) task-aware pruning, and 3) specialized model cascades-and evaluate their performance on a range of classification tasks. Moreover, for the first time, we investigate how these techniques complement one another, enabling up to 5× speedups with no loss in accuracy and 9.8× speedups while remaining within 2.5% of a highly accurate ResNet on specialized image classification tasks. These results suggest that simple and easy-to-implement specialization procedures may benefit a large number practical applications in which the representational power of general-purpose networks need not be inherited.
The authors review and evaluate several empirical methods to create faster versions of big neural nets for vision without sacrificing accuracy. They show using the ResNet architecture that combining distillation, pruning, and cascades are complementary and can yield pretty nice speedups.
This is a great idea and could be a strong paper, but it's really hard to glean useful recommendations from this for several reasons:
- The writing of the paper makes it hard to understand exactly what's being compared and evaluated. For a paper like this it's really crucial to be precise. When the authors say "specialization" or "specialized model", they sometimes mean distillation, sometimes filter pruning, and sometimes cascades. The distinction of "task-aware" also seems arbitrary to me and obfuscates the contribution of the paper as well. As far as I can tell, the technique is exactly the same, all that's changing is a slight modification. It's not like any of the intuitions or objectives are changing, so adding this new terminology just complicates things. For example, just say "We distill a parent model to a child model with a subset of the labels." 
- In terms of substance, the experiments don't really add much value in terms of general lessons. For example, the Cat/Dog from ImageNet distillation only works if the target labels are exactly a subset of the original. Obviously if the parent model was overcomplete before, it is certainly overcomplete now. The proposed cascade method is also fairly trivial -- a cheap distilled model backs off to the reference model. Why not train the whole cascade end-to-end? What about multiple levels of cascades? The only useful conclusion I can draw from the experiments is that (1) distillation still works (2) cascades also still work (3) pruning doesn't seem that useful in comparison. Training a cascade also involves a bunch of non-trivial design choices which are largely ignored -- how to set pass through, how to train the model, etc. etc.
- Nit: where are the blue squares in Figure 4? (Distill only) shouldn't those be the fastest methods (aside from pruning)? 
An ideal story would for a paper like this would be: here are some complementary ideas that we can combine in non-obvious ways for superlinear benefits, e.g. it turns out that by distilling into a cascade in some end-to-end fashion, you can get much better accuracy vs. speed trade-offs. Instead this paper is a grab-back of tricks. Such a paper can also provide value, but to do that right, the tricks need to be obvious *in retrospect only* and/or the experiments need to show a lot of precise practical lessons. All in all this paper reads like a tech report but not a conference publication.