Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners
<|reference_start|>Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners: Few-Shot Class Incremental Learning (FSCIL) is a task that requires a model to learn new classes incrementally without forgetting when only a few samples for each class are given. FSCIL encounters two significant challenges: catastrophic forgetting and overfitting, and these challenges have driven prior studies to primarily rely on shallow models, such as ResNet-18. Even though their limited capacity can mitigate both forgetting and overfitting issues, it leads to inadequate knowledge transfer during few-shot incremental sessions. In this paper, we argue that large models such as vision and language transformers pre-trained on large datasets can be excellent few-shot incremental learners. To this end, we propose a novel FSCIL framework called PriViLege, Pre-trained Vision and Language transformers with prompting functions and knowledge distillation. Our framework effectively addresses the challenges of catastrophic forgetting and overfitting in large models through new pre-trained knowledge tuning (PKT) and two losses: entropy-based divergence loss and semantic knowledge distillation loss. Experimental results show that the proposed PriViLege significantly outperforms the existing state-of-the-art methods with a large margin, e.g., +9.38% in CUB200, +20.58% in CIFAR-100, and +13.36% in miniImageNet. Our implementation code is available at https://github.com/KHU-AGI/PriViLege.<|reference_end|>
@article{park2024pre-trained,
    title={Pre-trained Vision and Language Transformers Are Few-Shot Incremental
  Learners},
    author={Keon-Hee Park, Kyungwoo Song, and Gyeong-Moon Park},
    journal={arXiv preprint arXiv:2404.02117},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.02117},
    primaryClass={cs.CV}
}