DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learning
<|reference_start|>DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learning: Cross-modal knowledge distillation (CMKD) refers to the scenario in which a learning framework must handle training and test data that exhibit a modality mismatch, more precisely, training and test data do not cover the same set of data modalities. Traditional approaches for CMKD are based on a teacher/student paradigm where a teacher is trained on multi-modal data with the aim to successively distill knowledge from a multi-modal teacher to a single-modal student. Despite the widespread adoption of such paradigm, recent research has highlighted its inherent limitations in the context of cross-modal knowledge transfer.Taking a step beyond the teacher/student paradigm, here we introduce a new framework for cross-modal knowledge distillation, named DisCoM-KD (Disentanglement-learning based Cross-Modal Knowledge Distillation), that explicitly models different types of per-modality information with the aim to transfer knowledge from multi-modal data to a single-modal classifier. To this end, DisCoM-KD effectively combines disentanglement representation learning with adversarial domain adaptation to simultaneously extract, foreach modality, domain-invariant, domain-informative and domain-irrelevant features according to a specific downstream task. Unlike the traditional teacher/student paradigm, our framework simultaneously learns all single-modal classifiers, eliminating the need to learn each student model separately as well as the teacher classifier. We evaluated DisCoM-KD on three standard multi-modal benchmarks and compared its behaviourwith recent SOTA knowledge distillation frameworks. The findings clearly demonstrate the effectiveness of DisCoM-KD over competitors considering mismatch scenarios involving both overlapping and non-overlapping modalities. These results offer insights to reconsider the traditional paradigm for distilling information from multi-modal data to single-modal neural networks.<|reference_end|>
@article{ienco2024discom-kd:,
    title={DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement
  Representation and Adversarial Learning},
    author={Dino Ienco (EVERGREEN, UMR TETIS, INRAE), Cassio Fraga Dantas (UMR
  TETIS, INRAE, EVERGREEN)},
    journal={British Machine Vision Conference, Nov 2024, Glasgow, United
  Kingdom},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.07080},
    primaryClass={cs.LG cs.AI cs.CV}
}