MST-Distill: Mixture of Specialized Teachers for Cross-Modal Knowledge
  Distillation
Knowledge distillation as an efficient knowledge transfer technique, has
achieved remarkable success in unimodal scenarios. However, in cross-modal
settings, conventional distillation methods encounter significant challenges
due to data and statistical heterogeneities, failing to leverage the
complementary prior knowledge embedded in cross-modal teacher models. This
paper empirically reveals two critical issues in existing approaches:
distillation path selection and knowledge drift. To address these limitations,
we propose MST-Distill, a novel cross-modal knowledge distillation framework
featuring a mixture of specialized teachers. Our approach employs a diverse
ensemble of teacher models across both cross-modal and multimodal
configurations, integrated with an instance-level routing network that
facilitates adaptive and dynamic distillation. This architecture effectively
transcends the constraints of traditional methods that rely on monotonous and
static teacher models. Additionally, we introduce a plug-in masking module,
independently trained to suppress modality-specific discrepancies and
reconstruct teacher representations, thereby mitigating knowledge drift and
enhancing transfer effectiveness. Extensive experiments across five diverse
multimodal datasets, spanning visual, audio, and text, demonstrate that our
method significantly outperforms existing state-of-the-art knowledge
distillation methods in cross-modal distillation tasks. The source code is
available at https://github.com/Gray-OREO/MST-Distill.
687918adcc15e42a72b01ae6
Hui Li
claimed_verified
2025-07-17T18:43:11.149Z
683fb1132548830f5d65fc6f
https://cdn-avatars.huggingface.co/v1/production/uploads/683fb1132548830f5d65fc6f/Xjasc-yrB0BdZpGKP4TVq.png
Hui Li
user
Gray1y
687918adcc15e42a72b01ae7
Pengfei Yang
687918adcc15e42a72b01ae8
Juanyang Chen
687918adcc15e42a72b01ae9
Le Dong
687918adcc15e42a72b01aea
Yanxin Chen
687918adcc15e42a72b01aeb
Quan Wang
MST-Distill: Mixture of Specialized Teachers for Cross-Modal Knowledge
  Distillation
Knowledge distillation as an efficient knowledge transfer technique, has
achieved remarkable success in unimodal scenarios. However, in cross-modal
settings, conventional distillation methods encounter significant challenges
due to data and statistical heterogeneities, failing to leverage the
complementary prior knowledge embedded in cross-modal teacher models. This
paper empirically reveals two critical issues in existing approaches:
distillation path selection and knowledge drift. To address these limitations,
we propose MST-Distill, a novel cross-modal knowledge distillation framework
featuring a mixture of specialized teachers. Our approach employs a diverse
ensemble of teacher models across both cross-modal and multimodal
configurations, integrated with an instance-level routing network that
facilitates adaptive and dynamic distillation. This architecture effectively
transcends the constraints of traditional methods that rely on monotonous and
static teacher models. Additionally, we introduce a plug-in masking module,
independently trained to suppress modality-specific discrepancies and
reconstruct teacher representations, thereby mitigating knowledge drift and
enhancing transfer effectiveness. Extensive experiments across five diverse
multimodal datasets, spanning visual, audio, and text, demonstrate that our
method significantly outperforms existing state-of-the-art knowledge
distillation methods in cross-modal distillation tasks. The source code is
available at https://github.com/Gray-OREO/MST-Distill.
MST-Distill, a novel cross-modal knowledge distillation framework, uses a mixture of specialized teachers and an instance-level routing network to address distillation path selection and knowledge drift, outperforming existing methods across multimodal datasets.
knowledge distillation cross-modal multimodal teacher models instance-level routing network knowledge drift masking module