  Knowledge distillation (KD), as an efficient and effective model compression
technique, has been receiving considerable attention in deep learning. The key
to its success is to transfer knowledge from a large teacher network to a small
student one. However, most of the existing knowledge distillation methods
consider only one type of knowledge learned from either instance features or
instance relations via a specific distillation strategy in teacher-student
learning. There are few works that explore the idea of transferring different
types of knowledge with different distillation strategies in a unified
framework. Moreover, the frequently used offline distillation suffers from a
limited learning capacity due to the fixed teacher-student architecture. In
this paper we propose a collaborative teacher-student learning via multiple
knowledge transfer (CTSL-MKT) that prompts both self-learning and collaborative
learning. It allows multiple students learn knowledge from both individual
instances and instance relations in a collaborative way. While learning from
themselves with self-distillation, they can also guide each other via online
distillation. The experiments and ablation studies on four image datasets
demonstrate that the proposed CTSL-MKT significantly outperforms the
state-of-the-art KD methods.

Machine Learning