  Considering the fact that students have different abilities to understand the
knowledge imparted by teachers, a multi-granularity distillation mechanism is
proposed for transferring more understandable knowledge for student networks. A
multi-granularity self-analyzing module of the teacher network is designed,
which enables the student network to learn knowledge from different teaching
patterns. Furthermore, a stable excitation scheme is proposed for robust
supervision for the student training. The proposed distillation mechanism can
be embedded into different distillation frameworks, which are taken as
baselines. Experiments show the mechanism improves the accuracy by 0.58% on
average and by 1.08% in the best over the baselines, which makes its
performance superior to the state-of-the-arts. It is also exploited that the
student's ability of fine-tuning and robustness to noisy inputs can be improved
via the proposed mechanism. The code is available at
https://github.com/shaoeric/multi-granularity-distillation.

Computer Vision and Pattern Recognition