Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models
<|reference_start|>Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models: Large-scale vision-language models (VLMs) have shown a strong zero-shot generalization capability on unseen-domain data. However, adapting pre-trained VLMs to a sequence of downstream tasks often leads to the forgetting of previously learned knowledge and a reduction in zero-shot classification performance. To tackle this problem, we propose a unique Selective Dual-Teacher Knowledge Transfer framework that leverages the most recent fine-tuned and the original pre-trained VLMs as dual teachers to preserve the previously learned knowledge and zero-shot capabilities, respectively. With only access to an unlabeled reference dataset, our proposed framework performs a selective knowledge distillation mechanism by measuring the feature discrepancy from the dual-teacher VLMs. Consequently, our selective dual-teacher knowledge distillation mitigates catastrophic forgetting of previously learned knowledge while preserving the zero-shot capabilities of pre-trained VLMs. Extensive experiments on benchmark datasets demonstrate that our framework is favorable against state-of-the-art continual learning approaches for preventing catastrophic forgetting and zero-shot degradation. Project page: https://chuyu.org/research/snd<|reference_end|>
@article{yu2024select,
    title={Select and Distill: Selective Dual-Teacher Knowledge Transfer for
  Continual Learning on Vision-Language Models},
    author={Yu-Chu Yu, Chi-Pin Huang, Jr-Jen Chen, Kai-Po Chang, Yung-Hsuan Lai,
  Fu-En Yang, Yu-Chiang Frank Wang},
    journal={arXiv preprint arXiv:2403.09296},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.09296},
    primaryClass={cs.CV}
}