Current knowledge distillation methods require full training data to distill knowledge from a large "teacher" network to a compact "student" network by matching certain statistics between "teacher" and "student" such as softmax outputs and feature responses. This is not only time-consuming but also inconsistent with human cognition in which children can learn knowledge from adults with few examples. This paper proposes a novel and simple method for knowledge distillation from few samples. Taking the assumption that both "teacher" and "student" have the same feature map sizes at each corresponding block, we add a 1 × 1 conv-layer at the end of each block in the student-net, and align the block-level outputs between "teacher" and "student" by estimating the parameters of the added layer with limited samples. We prove that the added layer can be absorbed/merged into the previous conv-layer to formulate a new conv-layer with the same size of parameters and computation cost as the previous one. Experiments verify that the proposed method is very efficient and effective to distill knowledge from teacher-net to student-net constructing in different ways on various datasets.
Model distillation can be tricky and in my own experience can take a lot of samples (albeit unlabeled, so cheaper and more readily available), as well as time to train. This simple trick seems to be doing quite well at training students quickly with few samples. However, it departs from most student-teacher training that find its primary purpose by actually outperforming students trained from scratch (on the full dataset without time constraints). This trick does not outperform this baseline, so its emphasis is entirely on quick and cheap. However, it's unclear to me how often that is actually necessary and I don't think the paper makes a compelling case in this regard. I am borderline on this work and could probably be swayed either way.
Strengths:
- It's a very simple and fast technique. As I will cover in a later bullet point (under weaknesses), the paper does not make it clear why this type of model distillation is that useful (since it doesn't improve the student model over full fine-tuning, unlike most student-teacher work). However, the reason why I do see some potential for this paper is because there might be a use case in quickly being able to adapt a pretrained network. It is very common to start from a pretrained model and then attach a new loss and fine-tune. Under this paradigm, it is harder to make architectural adjustments, since you are starting from a finite set of pretrained models made available by other folks (or accept the cost of re-training one yourself). However, it is unclear how careful one needs to treat the pretrained model if more fine-tuning is going to occur. If for instance you could just remove layers, drop some channels, glue it all together, and then that model would still be reasonable as a pretrained model since the fine-tuning stage could tidy everything up, then this method would not be useful in this situation.
- The fact that least squares solvers can be used at each stage, without the need for a final end-to-end fine-tune is interesting.
- It is good that the paper demonstrates improvements coupled with three separate compression techniques (Li et al., Liu et al., Guo et al.).
- The paper is technically thorough.
- It's good that the method is evaluated on different styles of networks (VGG, ResNet, DenseNet).
Weaknesses:
- Limited application because it only makes the distillation faster and cheaper. The primary goal of student-teacher training in literature is to outperform a student trained from scratch by the wisdom of the teacher. It ties into this notion that networks are grossly over-parameterized, but perhaps that is where the training magic comes from. Student-teacher training acknowledges this and tries to find a way to benefit from the over-parameterized training and still end up with a small model. I think the same motivation is used for work in  low-rank decomposition and many other network compression methods. However, in Table 1 the "full fine-tune" model is actually the clear winner and presented almost as an upper bound here, so the only benefit this paper presents is quick and cheap model distillation, not better models. Because of this, I think this paper needs to spend more time making a case for why this is so important.
- Since this technique doesn't outperform full fine-tuning, the goal of this work is much more focused on pure model compression. This could put emphasis on reducing model size, RAM usage reduction, or FLOPS reduction. The paper focuses on the last one, which is an important one as it correlates fairly well with power (the biggest constraint in most on-device scenarios). However, it would be great if the paper gave a broader comparison with compression technique that may have slightly different focus, such as low-rank decomposition. Size and memory usage could be included as columns in tables like 1, along with a few of these methods. 
- Does it work for aggressive compression? The paper presents mostly modest reductions (30-50%). I thin even if accuracy takes a hit, it could still work to various degrees. From what I can see, the biggest reduction is in Table 4, but FSKD is used throughout this table, so there is no comparison for aggressive compression with other techniques. 
- The method requires appropriate blocks to line up. If you completely re-design a network, it is not as straightforward as regular student-teacher training. Even the zero-student method requires the same number of channels at certain block ends and it is unclear from the experiments how robust this is. Actually, a bit more analysis into the zero student would be great. For instance, it's very interesting how you randomly initialize (let's say 3x3) kernels, and then the final kernels are actually just linear combinations of these - so, will they look random or will they look fairly good? What if this was done at the initial layer where we can visualize the filters, will they look smooth or not?
Other comments:
- A comparison with "Deep Mutual Learning" might be relevant (Zhang et al.). I think there are also some papers on gradually adjusting neural network architectures (by adding/dropping layers/channels) that are not addressed but seem relevant. I didn't read this recently, but perhaps "Gradual DropIn of Layers to Train Very Deep Neural Networks" could be relevant. There is at least one more like this that I've seen that I can't seem to find now.
- It could be more clear in the tables exactly what cited method is. For instance, in Table 1, does "Fine-tuning" (without FitNet/FSKD) correspond to the work of Li et al. (2016)? I think this should be made more clear, for instance by including a citation in the table for the correct row. Right now, at a glance, it would seem that these results are only comparing against prior work when it compares to FitNet, but as I read further, I understood that's not the case.
- The paper could use a visual aid for explaining pruning/slimming/decoupling.
Minor comments:
- page 4, "due to that too much hyper-parameters"
- page 4, "each of the M term" -> "terms"
- page 6, methods like FitNet provides" -> "provide"