  Knowledge distillation (KD) has shown very promising capabilities in
transferring learning representations from large models (teachers) to small
models (students). However, as the capacity gap between students and teachers
becomes larger, existing KD methods fail to achieve better results. Our work
shows that the `prior knowledge' is vital to KD, especially when applying large
teachers. Particularly, we propose the dynamic prior knowledge (DPK), which
integrates part of teacher's features as the prior knowledge before the feature
distillation. This means that our method also takes the teacher's feature as
`input', not just `target'. Besides, we dynamically adjust the ratio of the
prior knowledge during the training phase according to the feature gap, thus
guiding the student in an appropriate difficulty. To evaluate the proposed
method, we conduct extensive experiments on two image classification benchmarks
(i.e. CIFAR100 and ImageNet) and an object detection benchmark (i.e. MS COCO.
The results demonstrate the superiority of our method in performance under
varying settings. Besides, our DPK makes the performance of the student model
positively correlated with that of the teacher model, which means that we can
further boost the accuracy of students by applying larger teachers. More
importantly, DPK provides a fast solution in teacher model selection for any
given model.

Computer Vision and Pattern Recognition