REAL: Representation Enhanced Analytic Learning for Exemplar-free Class-incremental Learning
<|reference_start|>REAL: Representation Enhanced Analytic Learning for Exemplar-free Class-incremental Learning: Exemplar-free class-incremental learning (EFCIL) aims to mitigate catastrophic forgetting in class-incremental learning without available historical data. Compared with its counterpart (replay-based CIL) that stores historical samples, the EFCIL suffers more from forgetting issues under the exemplar-free constraint. In this paper, inspired by the recently developed analytic learning (AL) based CIL, we propose a representation enhanced analytic learning (REAL) for EFCIL. The REAL constructs a dual-stream base pretraining (DS-BPT) and a representation enhancing distillation (RED) process to enhance the representation of the extractor. The DS-BPT pretrains model in streams of both supervised learning and self-supervised contrastive learning (SSCL) for base knowledge extraction. The RED process distills the supervised knowledge to the SSCL pretrained backbone and facilitates a subsequent AL-basd CIL that converts the CIL to a recursive least-square problem. Our method addresses the issue of insufficient discriminability in representations of unseen data caused by a frozen backbone in the existing AL-based CIL. Empirical results on various datasets including CIFAR-100, ImageNet-100 and ImageNet-1k, demonstrate that our REAL outperforms the state-of-the-arts in EFCIL, and achieves comparable or even more superior performance compared with the replay-based methods.<|reference_end|>
@article{he2024real:,
    title={REAL: Representation Enhanced Analytic Learning for Exemplar-free
  Class-incremental Learning},
    author={Run He, Huiping Zhuang, Di Fang, Yizhu Chen, Kai Tong and Cen Chen},
    journal={arXiv preprint arXiv:2403.13522},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.13522},
    primaryClass={cs.LG cs.CV}
}