## Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning

Lu Yu, Zhe Tao, Hantao Yao, Member, IEEE, Joost Van de Weijer, and Changsheng Xu, Fellow, IEEE

Abstract -Deep neural networks (DNNs) excel on fixed datasets but struggle with incremental and shifting data in real-world scenarios. Continual learning addresses this challenge by allowing models to learn from new data while retaining previously learned knowledge. Existing methods mainly rely on visual features, often neglecting the rich semantic information encoded in text. The semantic knowledge available in the label information of the images, offers important semantic information that can be related with previously acquired knowledge of semantic classes. Consequently, effectively leveraging this information throughout continual learning is expected to be beneficial. To address this, we propose integrating semantic guidance within and across tasks by capturing semantic similarity using text embeddings. We start from a pre-trained CLIP model, employ the Semantically-guided Representation Learning (SG-RL) module for a soft-assignment towards all current task classes, and use the Semantically-guided Knowledge Distillation (SG-KD) module for enhanced knowledge transfer. Experimental results demonstrate the superiority of our method on general and fine-grained datasets. Our code can be found in https://github.com/aprilsveryown/semantically-guided-continual-learning.

Index Terms -Continual Learning, Vision-Language Models, Knowledge Transfer

✦

## 1 INTRODUCTION

A Large variety of algorithms were proposed to mitigate forgetting in continual learning, employing approaches such as rehearsal-based methods [1], [2], [3], [4], regularization-based techniques [5], [6], [7], and architecture-based solutions [8], [9], [10], [11], [12]. The emergence of Vision Transformers (ViTs) [13] has substantially enhanced the representation capabilities of pre-trained vision encoders for downstream tasks. A number of recent studies, including L2P [14] and DualPrompt [15], have leveraged the potential of pre-trained vision encoders in the context of continual learning, employing a prompt-based learning strategy. However, these works have exposed a notable limitation: performance degradation occurs when the pre-trained dataset exhibits a substantial semantic gap compared to the downstream data [16], [17].

Large scale pre-trained vision-language models, for instance ViT-BERT [18], CLIP [19] and UNITER [20], have emerged as powerful tools that combine computer vision and natural language processing, enabling machines to comprehend visual and textual information. These models can serve as strong foundation models for continual learning,

- · Lu Yu and Zhe Tao are with School of Computer Science and Engineering, Tianjin University of Technology, 300384, China (e-mail: luyu@email.tjut.edu.cn, tz@stud.tjut.edu.cn).
- · Hantao Yao is with State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China (e-mail: hantao.yao@nlpr.ia.ac.cn).
- · J. van de Weijer is with the Computer Vision Center, Universitat Autonoma de Barcelona, Barcelona 08193, Spain. (e-mail: joost@cvc.uab.es).
- · Changsheng Xu is the corresponding author, with State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China, and also with the School of Artificial Intelligence, University of the Chinese Academy of Sciences, Beijing, 100049, China (e-mail: csxu@nlpr.ia.ac.cn).

Manuscript received April 19, 2005; revised August 26, 2015.

as they were trained on massive datasets, allowing them to learn intricate patterns and relationships between images and language. As mentioned, the continual learning community has focused on applying foundational vision backbones [14], [15], [21], however, less effort has been invested in how to best exploit the rich semantic knowledge contained in the language encoders. A recent exception is Continual-CLIP [22] which shows impressive results in continual learning using both the vision and language encoder of the CLIP model. However, they keep the CLIP model frozen, thereby limiting the generalization to new tasks. In Fig. 1(a), we present an overview of several methods, each characterized by a distinct training strategy. The performance evaluations are shown in Fig. 1(b). The use of pre-trained vision encoders has resulted in incremental performance gains 1 . Interestingly, we will show that better exploitation of the knowledge contained in language encoders can lead to considerable performance improvements.

Establishing similarity between objects is a fundamental aspect of human perception and cognition, as highlighted by previous research [24]. pre-trained language models provide access to the semantic similarity between classes. In this paper, we propose two techniques to exploit the rich semantic information of pre-trained language models. Firstly, we exploit the intra-task semantic relationships between class labels. Instead of only using the available ground truth label to train on the newly arriving data, we replace the label with a soft-assignment towards all current task classes based on intra-task semantic relations. This ensures that each sample contributes to aligning the vision backbone with respect to

1. The methods L2P and DualPrompt obtain similar results without using any exemplars compared to existing state-of-the-art CL methods from scratch with exemplars (DER, DyTox++).

JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

| Method                                                                           | Language                                    | Vision                                                                      | Trainable                  |
|----------------------------------------------------------------------------------|---------------------------------------------|-----------------------------------------------------------------------------|----------------------------|
| DER [11] DyTox++ [9] L2P [14] DualPrompt [15] Continual-CLIP [22] CoOp [23] Ours | x x x x pre-trained pre-trained pre-trained | scratch scratch pre-trained pre-trained pre-trained pre-trained pre-trained | yes yes yes yes no yes yes |

Fig. 1: (a) Comparison of different models. (b) Performance comparison using different models across two datasets.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-2e78ff7c-c182-484a-86f8-a7d588feb7ae.jpg)

multiple semantic classes. Secondly, we propose to exploit the inter-task semantic relationships between old and new classes to improve stability during continual learning. For example, current class data for 'motor bikes' is expected to assign part of its prediction to the related previous task class of 'bikes' thereby preventing forgetting of previous classes. These inter-task relationships, derived from the language encoder, are used in a semantically-guided knowledge distillation technique.

To summarize, the main contributions of the paper are:

- · The usage of pre-trained models in continual learning has focused on pre-trained vision encoders. We show that exploiting the semantic information of the data labels with pre-trained language encoders can greatly enhance the accuracy of continual learning.
- · We extend the existing image distillation method based only on visual information with a term that exploits the knowledge of the labels of the current data with respect to the classes of previous tasks (inter-task semantic similarity). This improvement, called SG-KD , is shown to significantly increase the efficiency of distillation (and reduce catastrophic forgetting). In addition, we show how to exploit the intra-task semantic similarity (called SG-RL ).
- · Extensive experimental results show that our approach surpasses all methods by a substantial margin on several datasets under both many-shot and fewshot settings. Results on CIFAR100 under the 10-step setting show that the proposed method can greatly improve the accuracy after the last task with 11.4 points compared to the state-of-the-art.

## 2 RELATED WORK

## 2.1 Continual Learning

Continual learning research has witnessed significant advancements in recent years, resulting in the development of three main categories of approaches: rehearsal-based, regularization-based, and architecture-based. Rehearsalbased methods store previous task exemplars or generate fake exemplars using generative models to combat catastrophic forgetting. iCaRL [3] combines exemplar rehearsal with knowledge distillation, UCIR [2] addresses classification bias through normalization, and WA [4] ensures consistency in weight vector norms between old and new classes. Regularization-based methods impose constraints on network parameters, with strategies such as estimating parameter importance (EWC [5]) and distilling output from old and new models (LwF [6]) to retain information about previous tasks. SDC [25] proposed compensating the feature drift to prevent forgetting on some regularization-based continual learning methods. Architecture-based methods assign specified parameters to tasks or dynamically expand the network architecture. HAT [10] blocks and activates specific parts for old tasks, while DER [11] utilizes task-specific feature extractors and tackles parameter growth through pruning. Transformer-based continual learning paradigms, like DyTox [9], utilize shared self-attention layers and taskspecific tokens to achieve task-specialized embeddings. [26] proposed to apply Masked Autoencoders (MAEs) for continual learning and introduced a bilateral MAE framework that integrates learning from both image-level and embedding-level. [21] proposed to adaptively assign distillation loss weights by evaluating the relevance of each patch to the current task.

## 2.2 Continual Learning with Foundation Models

A recent emerging trend in the field of continual learning is to combine pre-trained vision transformers with parameterefficient fine-tuning techniques to continuously adapt the model to a stream of incoming downstream tasks. Specifically, these techniques involve prompt tuning [27], [28], prefix tuning [29], Adapter [30], [31], LoRA [32] , etc. The core of applying such a technique to continual learning is to construct additional learnable parameters or modules to instruct pre-trained representations and select appropriate prompts during inference time. L2P [14] optimizes the cosine similarity between query features and learnable keys, the most relevant prompts are selected and prepended to the token sequence both during training and inference. DualPrompt [15] further subdividing the prompts into general prompts sharing across all tasks and expert prompts picked and optimized in the same way as [14]. CODAPrompt [33] proposes to reweight prompts through input-conditioned weights, facilitating end-to-end optimization of the querykey mechanism across tasks. Distinct from the aforementioned prompt-based approaches, another line of work aims to construct classifiers through the extraction of robust feature representations via pretrained models. SLCA [17] finetuned the full pretrained model and recommended using different learning rates for the representation layers and the classifier to address the forgetting problem in representation

layers, they further incorporated prototype replay for posthoc alignment of the classification layers to reduce bias in the classifier. RanPAC [34] suggested employing a frozen random projection layer to project pretrained features into a high-dimensional space, thereby improving linear separability, they additionally utilize an online LDA classifier to eliminate correlations between categories. EASE [35] proposed to employ distinct adapters for each task to acquire task-specific features, and construct a unified classifier by synthesizing prototypes for old classes through semantic relevance.

## 2.3 Pre-trained Vision-Language Model

Large-scale vision-language pre-trained models like CLIP [19] and ALIGN [36] have emerged as powerful tools for representation learning due to their high transferability. Numerous studies [23], [37], [38] have focused on adapting these models to downstream tasks. For instance, CoOp [23] proposed vectorizing prompts, keeping the CLIP model fixed while optimizing only the prompt vectors. However, it was observed in [38] that prompts learned using this method exhibited poor generalization performance on unseen classes. To address this limitation, they introduced a meta-net to generate an input-conditional token for each image. Some works attempt to simultaneously optimize visual and textual prompts to improve the adaptability of vision-language models for downstream tasks. MaPLe [39] proposed to promote strong coupling between prompts of two different modalities to improve the consistency between visual and linguistic representations. PromptSRC [40] designed a regularization framework when training models on downstream tasks to prevent overfitting, which enhances the generalization capacity of the model. Differing from prompt-based approaches, some works try to incorporate adapters with MLPs into the transformer architecture to capture task-specific information. CLIPadapter [41] introduced extra bottleneck layers for learning new features. While these approaches have demonstrated significant improvements on various tasks compared to the adapted results provided by CLIP [19], they all shared the common limitation of freezing all the pre-trained parameters.

## 3 PRELIMINARY OF CLIP MODEL

Here we explain the details of the CLIP model. Let I denote the set of normalized image features encoded by the image encoder, and T denote the set of corresponding normalized text features. Assume that we have N samples, the logits for each image-text pair are computed as follows:

$$P _ { i, j } = \beta \cdot I _ { i } \cdot T _ { j } ^ { \top }, \quad \quad \ ( 1 ) \ \text{Affe}$$

where β is a learned scalar, P i,j is the cosine similarity score between the i th image and the j th text, and i ∈ { 1 , . . . , N } and j ∈ { 1 , . . . , N } are indices for the images and texts, re- spectively. Then the loss for each modality can be expressed as:

$$L _ { i m a g e } = - \frac { 1 } { N } \sum _ { \underset { \mathcal { S } } { i = 1 } } ^ { N } \log \left ( \frac { \exp ( P _ { i, i } ) } { \sum _ { j = 1 } ^ { N } \exp ( P _ { i, j } ) } \right ), \quad ( 2 )$$

$$L _ { t e x t } = - \frac { 1 } { N } \sum _ { j = 1 } ^ { N } \log \left ( \frac { \xi \exp ( P _ { j, j } ) } { \sum _ { i = 1 } ^ { N } \exp ( P _ { i, j } ) } \right ) \quad ( 3 )$$

The overall loss of the CLIP model is the sum of the image loss L image and text loss L text , which are averaged over the the number of samples. It can be represented as:

$$L _ { c } = \frac { 1 } { 2 } ( L _ { i m a g e } + L _ { t e x t } )$$

The loss function is minimized during pre-training to learn the joint representations of images and texts where semantically similar pairs are closer together and dissimilar pairs are farther apart.

The CLIP model has demonstrated remarkable continual learning performance without any fine-tuning [22]. To enhance its performance on new tasks and adapt to evolving data distributions over time, we thus propose to fine-tune the last block of the vision encoder. 2 We aim to leverage the strong generalization ability of the CLIP model while also updating its representation to better adapt to incoming data.

## 4 METHOD

To overcome the challenges of catastrophic forgetting during the training of pre-trained models, we introduce a novel approach that integrates semantic information guidance into the process of continual knowledge learning, as shown in Fig. 2. Leveraging the power of well-trained text embeddings, our proposed approach facilitates efficient interaction within and across task labels, leading to improved performance from two crucial perspectives. Firstly, we focus on learning more informative representations for new data by leveraging intra-task semantic similarity, thereby enhancing model plasticity ( SG-RL module). Secondly, we establish a relationship between old and new tasks by incorporating inter-task semantic similarity during the model distillation process, ensuring stability ( SG-KD module). In the following sections, we introduce these two components and explain how they work.

## 4.1 Intra-task Semantically-Guided Representation Learning (SG-RL)

As shown in the yellow part of Fig. 2, assume we have K classes contained in the current task t , their labels can be denoted as C t = { c t 1 , c t 2 , · · · , c t K } (for simplicity, we omit the superscript t in the following descriptions). We encode the text embeddings of these labels by the pre-trained language part of CLIP to obtain the normalized text embeddings T t = { T , T 1 2 , · · · , T K } , where || T k || = 1 and k ∈ { 1 2 , , · · · , K } . After obtaining the text embeddings of the current labels, we acquire the intra-task semantic similarity by computing

2. We fine-tune different parts of the vision encoder in Table 7. Our experiments indicate that fine-tuning the last block of the encoder results in the best performance.

Fig. 2: An overview of the proposed framework. We update the image encoder part of the CLIP model and train it with contrastive loss. We integrate semantic information into two modules: The SG-RL module, represented by the yellow block of the framework, aims to learn more discriminative representation to improve the model plasticity based on the intra-task semantic similarity. We train the model with KL-divergence loss between the generated semantically-guided labels and the original predictions. The SG-KD module, represented by the pink block of the framework, exploits the semantic similarity between current and previous task labels for knowledge distillation, to consolidate the model stability.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-15ec9261-de74-4c53-b270-a4fdf204e307.jpg)

the cosine similarity between each pair of text embeddings. This yields a text similarity matrix S t ↔ t which represents the intra-task similarity between the i -th and j -th labels; it can be described as follows:

where the logits P t = [ P t i,j ] N × K of N images in current task is computed by Eq. 1, and the prediction score ˆ y i,j = log exp( P t i,j ) ∑ K k =1 exp( P t i,k ) is after applying the log-softmax.

$$S _ { i, j } ^ { t \leftrightarrow t } = [ T _ { i } ^ { \top } T _ { j } ] _ { K \times K }, ( i, j \in \{ 1, 2, \cdots, K \} ), \quad ( 5 ) \quad \text{in Fig.}$$

This similarity score contains the pairwise similarity within each category within the current task.

To further convert the one-hot labels to our informative semantically-guided labels C sg = [ C sg i,j ] K × K , we compute the softmax function over each column of S t ↔ t with a parameter α which controls the degree of softness of the generated labels as follows:

$$C _ { i, j } ^ { s g } = \frac { \exp ( \alpha \cdot S _ { i, j } ^ { t \leftrightarrow t } ) } { \sum _ { k = 1 } ^ { K } \exp ( \alpha \cdot S _ { i, k } ^ { t \leftrightarrow t } ) } \quad \quad ( 6 ) \quad \text{tion} \quad \text{$4.2$}$$

The semantically-guided labels C sg encode the relationships and similarities between categories, allowing for a more comprehensive representation of class associations within the current task.

We then compute the KL-divergence loss L SG -RL between ˆ y i,j and the semantically-guided labels C sg i,j :

$$\mathcal { L } _ { S G - R L } = D _ { K L } ( \hat { y } | | C ^ { s g } ) = \sum _ { i = 1 } ^ { N } \sum _ { j = 1 } ^ { K } \hat { y } _ { i, j } \cdot \log \frac { \hat { y } _ { i, j } } { C ^ { s g } _ { i, j } }, \quad ( 7 ) \quad \text{are den}$$

The detailed illustration of the SG-RL model is shown in Fig. 3a. We propose to exploit the semantic knowledge of the pre-trained text-encoder by means of the intra-task semantic similarity between class labels (see Eq. 5). We replace the one-hot ground truth label in the cross entropy with a soft-assignment towards all current task classes. As a consequence, each sample contributes to aligning the vision backbone with respect to multiple semantic classes, leading to higher plasticity.

## 4.2 Inter-task Semantically-Guided Knowledge Distillation (SG-KD)

In this section, we aim to exploit the semantic knowledge of the pre-trained text-encoder to prevent forgetting and improve stability of the continual learner. Our main insight is that the semantic similarity between current and previous task labels can be exploited to prevent forgetting. For example, the current task label 'truck' can be used to prevent forgetting of previous related class labels like 'bus'. As shown Fig. 3b, given an image x i of the current task, the image features encoded by the previous image encoder are denoted as I t -1 i , and the corresponding normalized text embeddings of old-class labels y t -1 can be encoded by the

(a) Illustration of SG-RL module.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-eee4f22d-fe7a-44ac-8282-9ff1a3eebded.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-2f54d8a1-ddf9-4b55-8a36-d47f82bbc96d.jpg)

(b) Illustration of SG-KD module.

Fig. 3: An illustration of two components. (a) The SG-RL module calculates the text embedding similarity between the current labels using a pre-trained text encoder. It then generates semantically-guided labels that are more informative. (b) The SG-KD module constructs an inter-task similarity matrix based on text embeddings of previous and current task class labels. The most similar categories of the current class from old classes are selected by computing row-wise maximums (pink cells). The similarity value is added to the previous model's prediction logits at the most similar position, and the rest are subtracted from in each row. The enhanced logits are used for knowledge distillation between the current and previous models.

pre-trained text encoder as O j . Thus, the prediction logits P t -1 = [ P t -1 i,j ] N × M (the number of old classes is M ) of current data on the previous image encoder is computed as:

$$P _ { i, j } ^ { t - 1 } = \beta \cdot I _ { i } ^ { t - 1 } \cdot O _ { j } ^ { \top }, ( i \in 1, 2, \dots, N, j \in 1, 2, \dots, M ) \ ( 8 ) \ \text{adstillat}$$

The prediction logits of the current image encoder on oldclass heads P t = [ P t i,j ] N × M are:

where y t c i refers to the image label of image x i . The first term on the right-hand site is the same as in Eq. 11. The second term provides us the distillation between s y ( t -1 | y t c i ) and the current predictions. µ is the tradeoff between these two distillation losses. We derive s y ( t -1 | y t c i ) from the inter-task semantic similarity S t - ↔ 1 t between the text embeddings of current task labels 3 T and the old task labels O as follows:

$$P _ { i, j } ^ { t } = \beta \cdot I _ { i } ^ { t } \cdot O _ { j } ^ { \top },$$

which can be used to compute a prediction over the labels:

$$p ^ { t } ( y _ { j } ^ { t - 1 } | x _ { i } ) = \frac { \exp ( P _ { i, j } ^ { t } ) } { \sum _ { j } \exp ( P _ { i, j } ^ { t } ) } \quad \quad ( 1 0 ) \quad \text{class} \quad \text{the s} \quad \text{$\quad$}$$

and similarly for p t -1 ( y t -1 j | x i ) ; the difference is that this probability is based on the model at time t -1 . Traditional knowledge distillation [3], [6] between the old and current model can be described as follows:

$$\mathcal { L } _ { K D } = D _ { K L } ( p ^ { t } ( y ^ { t - 1 } | x _ { i } ), p ^ { t - 1 } ( y ^ { t - 1 } | x _ { i } ) ). \quad ( 1 1 )$$

Our contribution is that we aim to improve this distillation by also including the knowledge of the current sample labels about the previously seen classe in the distillation. Therefore, we use the semantic relationship between the current categories y t and the previous categories y t -1 , without requiring access to any visual information. We introduce Semantically-guided Knowledge Distillation (SG-KD); the loss for an image x i is given by:

$$\mathcal { L } _ { S G - K D } = & \ D _ { K L } ( p ^ { t } ( y ^ { t - 1 } | x _ { i } ), p ^ { t - 1 } ( y ^ { t - 1 } | x _ { i } ) ) \quad \text{where} \\ & + \mu D _ { K L } ( p ^ { t } ( y ^ { t - 1 } | x _ { i } ), s ( y ^ { t - 1 } | y ^ { t } _ { c _ { i } } ) ) \quad \text{(12)} \quad \text{$\text{used}$}$$

$$S _ { i, j } ^ { t - 1 \leftrightarrow t } = T _ { i } \cdot O _ { j } ^ { \top }$$

Here S t - ↔ 1 t i,j provides a measure of similarity between classes from different tasks. This allows the model to capture the semantic relationships between tasks and leverage this information for knowledge transfer. Thus, s y ( t -1 | y t c ) can be computed as:

$$s ( y _ { j } ^ { t - 1 } | y _ { c _ { i } } ^ { t } ) = \frac { \exp ( S _ { i, j } ^ { t - 1 \leftrightarrow t } / \tau ) } { \sum _ { j } \exp ( S _ { i, j } ^ { t - 1 \leftrightarrow t } / \tau ) } \quad \quad ( 1 4 )$$

τ is a hyper-parameter representing the temperature in the softmax function.

The final objective function can be described as follows:

$$\mathcal { L } = \mathcal { L } _ { c } + \lambda _ { 1 } \mathcal { L } _ { S G - R L } + \lambda _ { 2 } \mathcal { L } _ { S G - K D }, \quad \ \ ( 1 5 )$$

where λ 1 and λ 2 are trade-offs between the contrastive loss L c , KL-divergence loss L SG -RL and the semanticallyguided distillation loss L SG -KD .

TABLE 1: Details of all datasets used in this paper.

| Datasets                |   #Classes |   Train size |   Test size |
|-------------------------|------------|--------------|-------------|
| CIFAR100 [42]           |        100 |        50000 |       10000 |
| imagenet subset [43]    |        100 |       129395 |        5000 |
| miniImageNet [43]       |        100 |        50000 |       10000 |
| ImageNet1000 [43]       |       1000 |      1281167 |       50000 |
| Food-101 [44]           |        101 |        75750 |       25250 |
| Stanford Cars [45]      |        196 |         8144 |        8041 |
| FGVC Aircraft [46]      |        100 |         6667 |        3333 |
| Oxford-IIIT pets        |         37 |         3680 |        3669 |
| Caltech-101 [47]        |        102 |         3060 |        6085 |
| Oxford Flowers 102 [48] |        102 |         2040 |        6149 |
| CUB-200-2011 [49]       |        200 |         5994 |        5794 |
| Stanford Dogs [50]      |        120 |        12061 |        8519 |

## 5 EXPERIMENTS

## 5.1 Experimental Settings

## 5.1.1 Dataset and task spilt.

Our experimental evaluation begins by conducting experiments on three datasets that are commonly used for continual learning scenarios: CIFAR100 [42], imagenet subset, and ImageNet1000 [43]. For CIFAR100, we adopt the incremental phase splitting into 10, 20, and 50 steps, following the approach of DyTox [9]. In each split, we further assess the performance of the proposed method under three different class orders. Regarding imagenet subset and ImageNet1000, we split the incremental phase into 10 steps, with 10/100 new classes added at each incremental step. We further evaluate the performance on eight fine-grained datasets. We also evaluate for few-shot continual learning on miniImagenet, and CUB-200-2011 following [52], [53]. For miniImagenet, we split the 100 classes into 60 base classes and 40 classes across 8 sessions with 5 new classes per session. CUB-200-2011 is divided into 100 base classes and 100 classes across 10 sessions with 10 new classes per session. Each incremental session for all three datasets consists of 5 training samples per new class.

The dataset splits follow two patterns: in the A + B × C split, A represents the number of classes in the initial task, C indicates the total number of steps, and B signifies the number of new classes added at each incremental stage; for the A × B split, A new classes are introduced in each of the B incremental steps. Note that we handle the OxfordIIIT pets dataset separately due to its 12 cat categories and 25 dog categories, splitting it into two stages: one for cats and another for dogs. The details of all datasets are listed in Table 1.

## 5.1.2 Implementation details.

All experiments were conducted with a vision backbone of ViT-B/16 version CLIP model. We train every task for 10 epochs except for CUB and Aircrafts with 20 epochs. We adopt SGD in all experiments as the optimizer with an initial learning rate of 0.01(0.001 for few-shot setting), weight decay is 2e-4 and 0.9 for momentum. Batch size is 256 for all experiments. We set α to 13 in Eq. 6, β is set to 100 in Eq. 4.2 following [22] and τ to 0.1 in Eq. 14 for all datasets. The trade-off parameters, λ 1 and λ 2 , are set to 0.5 and 0.1 across

3. The text embeddings of exemplar labels are integrated into T i , when applicable.

all datasets (the hyper-parameters sensitive experiments can be found in Section 5.4.5 ). We save 20 exemplars for each old class by herding algorithm in all experiments except the ablation on different exemplar sizes (in Section 5.4.4), following the setting in [3], [9], [11]. Under the few-shot setting, none exemplar is saved following the setting in [52], [53]. To ensure fairness, methods trained with a pre-trained model are initialized with CLIP . For implementing 'Linear 4 Probe', we directly add a fully connected layer after the linear projection, we train 100 epochs for each dataset with an initial learning rate of 0.1 and decays by 0.1 every 45 epochs. For 'Joint', to ensure fairness in comparison, we fine-tune the last block of the network for joint training, all training samples are acquired concurrently in the same session. We train 100 epochs for each dataset with an initial learning rate of 0.01 and decays by 0.1 every 45 epochs.

For all experiments in the paper, we applied RandomResizedCrop, RandomHorizontalFlip and normalization for data augmentation. The random seed was set to 1993 except when spliting CIFAR100 with three different class orders as shown in Table 2. We use randomly generated integers ranging from 1 to 5000 for the other two class orders.

Test-time evaluation details: For each test sample x test , the image is passed through the image encoder, resulting in its image feature I test . Meanwhile, we collect all previously seen categories as prompts and feed them to the text encoder to obtain the text features T = { T i ; 1 ≤ i ≤ k } , where k is the number of seen classes. Then we compute the cosine similarity between image feature and text features:

$$s _ { i } = \frac { I _ { t e s t } \cdot T _ { i } } { \| I _ { t e s t } \, \| \cdot \| \, T _ { i } \| }$$

where 1 ≤ i ≤ k , the prediction score for x test can then be computed as:

$$p = \frac { \exp ( s _ { i } ) } { \sum _ { i = 1 } ^ { k } \exp ( s _ { i } ) }$$

The class with the highest predicted score is considered as the predication.

## 5.2 Full-shot Continual Learning Setting

## 5.2.1 Evaluation on general datasets.

We conducted extensive experiments on three benchmark datasets, namely CIFAR100, imagenet subset, and ImageNet1000 in comparison to state-of-the-art approaches. In Table 2, we present the averaged accuracy on CIFAR100 over three different class orders and three different splits. In the most common 10-step split, our method achieves an impressive accuracy of 80.1 after the last incremental step, surpassing the previous state-of-the-art method by 11.4 points. Even for longer task sequences of 20 or 50 steps, our method consistently maintains superior performance, with at least 9.5 and 5.4 points higher accuracy than the state-ofthe-art 'Last' result respectively. Moving to larger-scale continual learning scenarios, as shown in Table 3, our method demonstrates even greater superiority. On imagenet subset

4. Since the exact prompt and specific class names employed by CLIP [19] for each dataset are unknown. We tried several prompts and finally for each dataset we utilize a specific prompt so that the results of our own implementation are comparable or identical to those provided by CLIP [19].

TABLE 2: The results on CIFAR100 are averaged over three class orders (seeds) under 10, 20, and 50 steps, following the setting proposed in [9]. The bold parts represent the best results, underlined parts indicate the second best, and up arrows show the improvement of the best over the second best. Re-implemented results are marked with an asterisk (*) .

| CIFAR100            | 10 steps   | 10 steps      | 10 steps       | 20 steps   | 20 steps      | 20 steps      | 50 steps   | 50 steps      | 50 steps      |
|---------------------|------------|---------------|----------------|------------|---------------|---------------|------------|---------------|---------------|
| Methods             | #Param.    | Avg           | Last           | #Param.    | Avg           | Last          | #Param.    | Avg           | Last          |
| Linear probe        | -          | -             | 83.1           | -          | -             | 83.1          | -          | -             | 83.1          |
| Joint               | 7.09       | -             | 83.9           | 7.09       | -             | 83.9          | 7.09       | -             | 83.9          |
| UCIR [2]            | 11.22      | 58.7          | 43.4           | 11.22      | 58.2          | 40.6          | 11.22      | 56.9          | 37.1          |
| BiC [51]            | 11.22      | 68.8          | 53.5           | 11.22      | 66.5          | 47.0          | 11.22      | 62.1          | 41.0          |
| WA [4]              | 11.22      | 69.5          | 53.8           | 11.22      | 67.3          | 47.3          | 11.22      | 64.3          | 42.1          |
| PODNet [8]          | 11.22      | 58.0          | 41.1           | 11.22      | 54.0          | 35.0          | 11.22      | 51.2          | 33.0          |
| DER w/o p [11]      | 112.27     | 75.4          | 65.2           | 224.55     | 74.1          | 62.5          | 561.39     | 72.4          | 59.1          |
| DER [11]            | -          | 74.6          | 64.4           | -          | 74.0          | 62.6          | -          | 72.1          | 59.8          |
| DyTox++ [9]         | 10.73      | 77.0          | 67.5           | 10.74      | 76.8          | 64.3          | 10.77      | 75.5          | 59.5          |
| Continual-CLIP [22] | -          | -             | 68.7           | -          | -             | 68.7          | -          | -             | 68.7          |
| CoOp ∗ [23]         | 7.91       | 76.2          | 68.1           | 7.91       | 77.0          | 67.6          | 7.91       | 78.3          | 66.3          |
| CoCoOp ∗ [38]       | 7.13       | 75.1          | 65.9           | 7.13       | 75.1          | 63.1          | 7.13       | 74.8          | 61.1          |
| L2P ∗ [14]          | 7.38       | 76.3          | 65.9           | 7.38       | 75.2          | 66.2          | 7.38       | 76.5          | 64.6          |
| DualPrompt ∗ [15]   | 7.42       | 71.3          | 65.1           | 7.42       | 66.2          | 65.4          | 7.42       | 70.1          | 66.6          |
| Ours                | 7.09       | 86.6 ( ↑ 9.6) | 80.1 ( ↑ 11.4) | 7.09       | 86.0 ( ↑ 9.0) | 78.2 ( ↑ 9.5) | 7.09       | 81.8 ( ↑ 3.5) | 74.1 ( ↑ 5.4) |

TABLE 3: We compare the results obtained on the imagenet subset dataset (100 classes) and the ImageNet full dataset (1000 classes), where the training session is split into 10 steps, with each step containing an equal number of classes for each task.

|                     | imagenet subset 10 steps   | imagenet subset 10 steps   | imagenet subset 10 steps   | imagenet subset 10 steps   | imagenet subset 10 steps   | ImageNet1000 10 steps   | ImageNet1000 10 steps   | ImageNet1000 10 steps   | ImageNet1000 10 steps   | ImageNet1000 10 steps   |
|---------------------|----------------------------|----------------------------|----------------------------|----------------------------|----------------------------|-------------------------|-------------------------|-------------------------|-------------------------|-------------------------|
| Methods             | #Param.                    | top-1                      | top-1                      | top-5                      | top-5                      | #Param.                 | top-1                   | top-1                   | top-5                   | top-5                   |
|                     | #Param.                    | Avg                        | Last                       | Avg                        | Last                       | #Param.                 | Avg                     | Last                    | Avg                     | Last                    |
| Linear probe        | -                          | -                          | 83.9                       | -                          | 97.6                       | -                       | -                       | 80.2                    | -                       | 94.1                    |
| Joint               | 7.09                       | -                          | 86.1                       | -                          | 98.3                       | 7.09                    | -                       | 81.1                    | -                       | 96.5                    |
| DER w/o p [11]      | 112.27                     | 77.2                       | 66.7                       | 93.2                       | 87.5                       | 116.89                  | 68.8                    | 60.2                    | 88.2                    | 82.9                    |
| DER [11]            | -                          | 76.1                       | 66.1                       | 92.8                       | 88.4                       | -                       | 66.7                    | 58.6                    | 87.1                    | 81.9                    |
| DyTox++ [9]         | 11.01                      | 80.8                       | 72.5                       | 94.4                       | 90.1                       | -                       | -                       | -                       | -                       | -                       |
| Continual-CLIP [22] | -                          | -                          | 75.2                       | -                          | 96.9                       | -                       | -                       | 68.6                    | -                       | 90.6                    |
| CoOp ∗ [23]         | 7.91                       | 80.9                       | 70.2                       | 97.5                       | 94.5                       | 15.28                   | 75.2                    | 66.7                    | 95.9                    | 91.9                    |
| CoCoOp ∗ [38]       | 7.13                       | 81.8                       | 71.0                       | 98.5                       | 96.1                       | 7.13                    | 64.6                    | 56.0                    | 92.3                    | 88.7                    |
| L2P ∗ [14]          | 7.38                       | 82.2                       | 72.7                       | 98.5                       | 96.4                       | 8.07                    | 71.9                    | 63.3                    | 95.0                    | 90.4                    |
| DualPrompt ∗ [15]   | 7.42                       | 79.6                       | 68.4                       | 97.7                       | 95.0                       | 8.11                    | 70.7                    | 64.0                    | 94.4                    | 92.1                    |
| Ours                | 7.09                       | 89.8 ( ↑ 7.6)              | 83.1 ( ↑ 7.9)              | 99.1 ( ↑ 0.6)              | 98.4 ( ↑ 1.5)              | 7.09                    | 83.4 ( ↑ 8.2)           | 75.1 ( ↑ 6.5)           | 97.4 ( ↑ 1.5)           | 95.2 ( ↑ 3.1)           |

TABLE 4: Results on eight fine-grained datasets with different task splits.

| Dataset             | Food           | Cars          | Aircraft      | Pets     | Caltech       | Flowers    | CUB           | Dogs          |
|---------------------|----------------|---------------|---------------|----------|---------------|------------|---------------|---------------|
| #classes split      | 101 11+ 9 × 10 | 196 28 × 7    | 100 10 × 10   | 37 12+25 | 102 17 × 6    | 102 17 × 6 | 200 20 × 10   | 120 12 × 10   |
| Linear probe Joint  | 92.8 93.1      | 86.7          | 59.5          | 93.1     | 94.7          | 98.1       | 80.4 ∗        | 78.6 ∗        |
|                     |                | 91.5          | 73.2          | 94.7     | 93.8          | 98.1       | 85.0          | 80.6          |
| Continual-CLIP [22] | 89.2           | 65.6          | 27.1          | 88.9     | 89.3          | 70.4       | 55.6 ∗        | 63.4 ∗        |
| CoOp ∗ [23]         | 84.6           | 86.1          | 57.6          | 94.0     | 90.2          | 97.0       | 80.1          | 75.8          |
| CoCoOp ∗ [38]       | 79.1           | 84.9          | 59.8          | 92.8     | 88.2          | 97.2       | 77.0          | 69.7          |
| L2P ∗ [14]          | 81.7           | 86.9          | 61.1          | 92.8     | 88.6          | 97.0       | 82.6          | 75.9          |
| DualPrompt ∗ [15]   | 80.5           | 82.7          | 45.5          | 90.6     | 86.4          | 96.7       | 79.6          | 70.3          |
| Ours                | 90.9 ( ↑ 1.7)  | 88.6 ( ↑ 2.5) | 66.6 ( ↑ 5.5) | 94.0     | 92.3 ( ↑ 2.1) | 96.2       | 83.7 ( ↑ 1.1) | 79.6 ( ↑ 3.7) |

dataset, we outperform the state-of-the-art result by 7.9 points in terms of top-1 'Last' accuracy, approaching the performance of linear probing [19]. Furthermore, on the challenging ImageNet1000 dataset, our method achieved significant improvements with a 6.5-point increase in top1 'Last' accuracy and a 3.1-point increase in top-5 accuracy. Overall, our experimental results demonstrate that the proposed method consistently outperforms state-of-the-art approaches on all three datasets, showcasing its superior performance and stability in continual learning settings.

## 5.2.2 Evaluation on fine-grained datasets.

To further validate the effectiveness of the proposed method, we conduct experiments on eight fine-grained datasets. In fine-grained datasets, all categories belong to the same parent category in a hierarchical relationship, which intuitively indicates a high degree of semantic similarity among them, which is more challenging. In Table 4, we present the performance of our proposed method compared to other methods on different splits of the datasets. We notice that the Continual-CLIP evaluation on datasets like Cars, Aircrafts, and CUB, is initially poor. State-of-the-art methods obtain a substantial improvement, particularly on Aircraft where the gain reaches 34 points with L2P. Our method achieves performance very close to the joint training results, with a gap of less than 1 point on multiple datasets such as Pets and Dogs. Moreover, our method even surpasses the results of linear probing evaluation on multiple datasets, which is typically used to assess the quality of pretrained features.

Fig. 4: The accuracy change during few-shot incremental training sessions on CUB-200-2011 and miniImageNet datasets.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-8769b1b7-f064-45cd-93a4-d51c99dbb1ba.jpg)

TABLE 5: The effect of different components. 'FT' represents only applying contrastive loss for training, 'One-hot Label' utilizes the prompt of each class as a classification head during training, and uses the one-hot encoded labels to calculate the cross-entropy loss. 'Naive KD' distills the output of the current data between old and new models with the KD loss proposed in [54] (ref Eq. 11).

|             | FT        | One-hot Label   | SG-RL(Eq. 6)   | Naive KD(Eq. 11)   | SG-KD(Eq. 12)   | CIFAR100 10 steps        | imagenet subset 10 steps   |
|-------------|-----------|-----------------|----------------|--------------------|-----------------|--------------------------|----------------------------|
| C-CLIP [22] |           |                 |                |                    |                 | 68.7                     | 75.2                       |
|             | ✓ ✓ ✓ ✓ ✓ | ✓               | ✓ ✓ ✓          | ✓                  | ✓               | 75.7 71.4 77.0 77.7 80.6 | 81.1 75.2 81.8 82.6 83.1   |

## 5.3 Few-shot Continual Learning Setting

Our proposed method has demonstrated impressive results on several datasets, including coarse and fine-grained datasets. It is interesting to evaluate its effectiveness under the few-shot continual learning setting, which poses a significant challenge. In this setting, the model needs to quickly adapt to new classes with limited labeled data while preserving knowledge of previous tasks. We present dynamic accuracy curves that showcase the incremental training sessions on the CUB-200-2011 and miniImageNet datasets in Fig. 4. The curves highlight the notable performance of our method (shown in red) as it significantly outperforms state-of-the-art approaches with (in solid line) and without (in dashed line) ViT backbone by a substantial margin over all incremental sessions. Particularly on the miniImageNet dataset, our method achieves a final accuracy about 36.5 points higher than FACT and 5.3 points higher than CoOp.

## 5.4 Ablation Study

## 5.4.1 Components of our proposed method

We conduct ablation experiments on CIFAR100 and imagenet subset to thoroughly examine the effect of the different proposed modules. Both datasets were split into 10 steps, and the results are shown in Table 5. On both datasets, the contrastive loss significantly improves the final accuracy compared to Continual-CLIP. However, when we additionally apply the one-hot encoded labels to add an auxiliary classification loss, we observe a significant performance drop. By incorporating our 'SG-RL' module, the model shows a gain of 1.3 points on CIFAR100 and 0.7

TABLE 6: Comparison of different vision models on CIFAR100 and imagenet subset, learned with 10 steps.

|           |             | CIFAR100   | CIFAR100   | imagenet subset   | imagenet subset   |
|-----------|-------------|------------|------------|-------------------|-------------------|
|           |             | Avg        | Last       | Avg               | Last              |
| ResNet 50 | C-CLIP [22] | -          | 41.6       | -                 | 65.6              |
|           | Ours        | 59.8       | 50.7       | 80.2              | 71.3              |
| ViT-B/16  | C-CLIP [22] | -          | 68.7       | -                 | 75.2              |
|           | Ours        | 86.6       | 80.1       | 89.8              | 83.1              |
| ViT-L/14  | C-CLIP [22] | -          | 77.9       | -                 | 81.5              |
|           | Ours        | 91.1       | 86.5       | 91.9              | 85.8              |

points on imagenet subset. Furthermore, when we apply knowledge distillation to the loss function, our proposed 'SG-KD' demonstrates superior performance over 'Naive KD' (ref Eq. 11) on both datasets, with a particularly notable gain of 2.9 points on Cifar100.

## 5.4.2 Validation on various vision models

Our primary experiments employed ViT-B/16 as the image encoder. However, it is important to investigate the effectiveness of our method with other architectures as well. Hence, we validate our approach on two additional vision models: ResNet 50 and ViT-L/14. Table 6 presents the comparison between Continual-CLIP (illustrated as 'C-CLIP' for short in the table) evaluation and our proposed method on CIFAR100 and imagenet subset within 10 steps. Despite the differences in architecture and parameter count, all three image encoders exhibit significant improvements compared to Continual-CLIP evaluation. These results suggest that our method is effective across different scales of image encoders, irrespective of architecture.

## TABLE 7: Comparison of training different layers of image encoder.

Fig. 5: The comparison of accuracy when saving different number of exemplars on CIFAR100 and imagenet subset.

|                 |         | Last 6 blocks   | Last 3 blocks   | Proj. + last block   | Proj.     | Last block   |
|-----------------|---------|-----------------|-----------------|----------------------|-----------|--------------|
| CIFAR100        | FT Ours | 58.4 68.9       | 73.2 79.8       | 75.2 80.1            | 74.6 77.8 | 75.7 80.6    |
| imagenet subset | FT Ours | 58.3 75.6       | 80.2 82.7       | 79.6 81.5            | 79.9 79.7 | 81.1 83.1    |

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-71b40398-dbf3-44f3-9c59-37af44727e25.jpg)

TABLE 8: Ablation study on hyper-parameters sensitivity. ('C' stands for 'CIFAR100' and 'I' stands for 'imagenet subset'.)

## 5.4.5 Hyper-parameters sensitivity

| λ 1   |   0.1 |   0.5 |    1 |   0.1 |   0.5 |    1 |   0.1 |   0.5 |    1 |
|-------|-------|-------|------|-------|-------|------|-------|-------|------|
| λ 2   |   0.1 |   0.1 |  0.1 |   0.5 |   0.5 |  0.5 |   1   |   1   |  1   |
| C     |  80   |  80.6 | 80.5 |  79.6 |  79.8 | 79.6 |  78.5 |  78.6 | 78.4 |
| I     |  82.7 |  83.1 | 82.5 |  80.7 |  81.2 | 81.1 |  79.7 |  79.6 | 79.5 |

## 5.4.3 Effect of training different layers of image encoder

From Table 7, we observe that when training the last 6 blocks of the image encoder (half of the blocks in ViTB/16), the model fails to overcome catastrophic forgetting. However, significant improvements are achieved when our proposed SG-RL and SG-KD modules are applied. Training only the linear projection layer yields decent results, but it appears that the model's potential is greatly limited with such a small number of parameters. To explore the impact of incorporating more parameters into the training process, we gradually training additional blocks. It is evident that the best performance is achieved when only the last blocks are trained.

## 5.4.4 Comparison of different exemplar sizes

Some previous works [55], [56] have shown that saving a certain number of exemplars for old classes is very useful for defying forgetting. It can be seen from Fig 5, the accuracy exhibits a gradual increase as the number of saved exemplars increases. Our proposed method improves fine-tuning results in all cases. It is worth mentioning that on ImageNet subset, performance of fine-tuning with only one exemplar for each old class drops a lot compared with Continual-CLIP evaluation, While our method still improves compared to Continual-CLIP, and shows a huge gap with fine-tuning.

We conduct sensitivity analysis experiments on the two proposed losses to assess their impact. It can be seen from Table 8, our method shows strong robustness to both hyperparameters ranged from 0.1 to 1. The best results are obtained when λ 1 and λ 2 are set to 0.5, 0.1 respectively.

## 5.5 More Analysis

## 5.5.1 Results with frozen backbone

Given that the methods we compared are all proposed based on frozen backbone while our method proposes to fine-tune the last block of the vision encoder. For a fair comparison, in the above experiment section we fine-tuned the same part of parameters for these methods. Here for a more comprehensive and in-depth comparison, we present the experimental results of these methods in the case of frozen backbone in Table 9. It can be clearly seen that adapting the backbone outperforms the frozen backbone by a large margin in vast majority of cases. It is particularly worth mentioning that on Out-Of-Distribution tasks such as Aircrafts and Cars, most methods show poor performance in the case of frozen backbone, indicating that these methods struggle to adapt to these tasks with only prompt steering pre-trained representations.

## 5.5.2 Comparison with one-hot label

We adopted contrastive loss as the baseline performance for fine-tuning in the main paper, here we investigated the naive cross-entropy loss by one-hot Label. We compute the cross-entropy loss between the predictions of the network on the current classes and the ground-truth, and use the prompt of each class as a classification head. As it can be seen from the Table 10 and Table 11, fine-tuning by only

| Dataset              | CIFAR100    | imagenet subset   | ImageNet1000   | Food           | Cars       | Aircraft    | Pets      | Caltech    | Flowers    | CUB         | Dogs        |
|----------------------|-------------|-------------------|----------------|----------------|------------|-------------|-----------|------------|------------|-------------|-------------|
| #classes split       | 100 10 × 10 | 100 10 × 10       | 1000 100 × 10  | 101 11+ 9 × 10 | 196 28 × 7 | 100 10 × 10 | 37 12 +25 | 102 17 × 6 | 102 17 × 6 | 200 20 × 10 | 120 12 × 10 |
| Linear probe 5 Joint | 83.1 83.9   | 83.9 86.1         | 80.2 81.1      | 92.8 93.1      | 86.7 91.5  | 59.5 73.2   | 93.1 94.7 | 94.7 93.8  | 98.1 98.1  | 80.4 ∗ 85.0 | 78.6 ∗ 80.6 |
|                      | 68.7        | 75.2              | 68.6           | 89.2           | 65.6       |             |           |            |            | 55.6 ∗      | 63.4 ∗      |
| Continual-CLIP [22]  |             |                   |                |                |            | 27.1        | 88.9      | 89.3       | 70.4       |             |             |
| CoOp [23]            | 66.6        | 72.0              | 67.5           | 84.7           | 81.7       | 48.4        | 93.3      | 86.9       | 94.2       | 78.1        | 74.0        |
| CoCoOp [38]          | 64.1        | 69.1              | 58.9           | 86.0           | 74.9       | 39.4        | 93.2      | 86.6       | 93.0       | 70.0        | 69.7        |
| L2P [14]             | 57.8        | 59.7              | 58.8           | 74.9           | 63.6       | 27.3        | 83.3      | 73.1       | 77.6       | 70.0        | 58.9        |
| DualPrompt [15]      | 67.4        | 71.3              | 65.6           | 85.2           | 80.9       | 49.9        | 90.5      | 90.5       | 96.8       | 76.2        | 69.8        |
| Ours                 | 80.6        | 83.1              | 75.1           | 90.9           | 88.6       | 66.6        | 94.0      | 92.3       | 96.2       | 83.7        | 79.6        |

TABLE 9: Results with frozen backbone on all datasets.

| Dataset            | Food       | Cars      | Aircraft   | Pets      | Caltech   | Flowers   | CUB       | Dogs      |
|--------------------|------------|-----------|------------|-----------|-----------|-----------|-----------|-----------|
| split              | 11+ 9 × 10 | 28 × 7    | 10 × 10    | 12+25     | 17 × 6    | 17 × 6    | 20 × 10   | 12 × 10   |
| One-hot Label Ours | 83.9 90.8  | 85.2 88.6 | 52.4 66.6  | 92.2 94.0 | 90.2 92.3 | 92.4 96.2 | 77.4 83.7 | 72.9 79.6 |

TABLE 10: Results compared to one-hot label on eight fine-grained datasets with different task splits.

Fig. 6: T-SNE visualization on test data with Continual-CLIP (left two) and our method (right two).

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-edcb33ca-f991-4fe0-8763-ffb5e6954c5c.jpg)

TABLE 11: Results compared to one-hot label on three general datasets with different task splits.

| Dataset            | CIFAR100   | imagenet subset   | ImageNet1000   |
|--------------------|------------|-------------------|----------------|
| split              | 10 × 10    | 10 × 10           | 100 × 10       |
| One-hot Label Ours | 71.2       | 73.2              | 64.3           |
|                    | 80.6       | 83.1              | 75.1           |

the cross-entropy loss calculated by one-hot label on some datasets achieve reasonable results, but it still shows a large gap compared with our method. To some extent this also reflects that only learning the image-to-label mapping tends to suffer from more severe catastrophic forgetting during incremental learning process.

## 5.6 Visualization

the right side shows the results obtained by our proposed method. For the new task ('task1'), Continual-CLIP (the first) can somewhat separate the classes, but it is evident that samples belonging to the same class are scattered and not well-clustered. Conversely, our method (the third) successfully clusters each class and significantly increases the distance between classes. When all data is mixed ('task0 &amp; task1'), it becomes apparent that Continual-CLIP (the second) lacks clear and meaningful boundaries between classes. Each class occupies a large space, indicating poor intra-class clustering. In contrast, ours clearly separates both the old and new tasks (the fourth). Notably, the categories 'airplane' , 'truck' and 'ship' are positioned far away from the animal categories such as 'horse' , 'dog' , 'deer' , and 'cat' . This demonstrates that our method, with its semanticallyguidance, better understands the semantic meaning of the categories and capture their relationships.

## 5.6.1 Visualization of feature representation

For the sake of simplicity and intuitiveness, we utilize the CIFAR10 dataset as an example to visualize the t-SNE representation [57]. We split the dataset into two steps, with each containing 5 new classes. The two sub-figures on the left shown in Fig. 6 represent the t-SNE visualization of the feature representation evaluated on Continual-CLIP, while

## 5.6.2 Heat Map Visualization

We present the heat maps visualization in Fig. 7 as it was proposed in [58]. As can be seen from the 'vulture' in the first row, zero-shot CLIP seems to have focused part of its attention on the wood and bushes, which may indicate that it mistook these two parts for vultures, considering the

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-357c5272-f8cc-42c1-a3cc-578e914d9bf8.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-740505a3-6209-4d23-aa6f-f1a8d5b66f81.jpg)

zero-shot

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-88a95915-f92c-40fb-bd56-1fff3426a82c.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-6cf5004a-2985-4931-8398-55ec32330b9d.jpg)

ours

Fig. 7: The heat map visualization comparison of zero-shot and our method.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-2e50e766-a08d-40d7-ac47-b85bb66c9aa5.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-c022cc7f-53b0-4d30-8309-8d96cc763528.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-e36243ed-3325-4119-80fb-a7b1cdc50340.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-d71d38e2-f714-4bf0-abcb-d6eef0b9b2eb.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01076v1-figure-5bad803c-29cf-407c-9174-8984cccbc2a2.jpg)

color of the wood and the vultures are very similar. After applying our method, the attention is completely focused on the vulture, while the rest of the noise is completely eliminated. As for the 'peacock' in the second row, it is obvious that the attention completely covers the entire peacock after applying our method. And for the 'hummingbird' in the last row, the zero-shot CLIP seems to confuse the bird in the upper left corner, small part of the water bottle, and the hummingbird itself, while our method only focuses the attention completely on the hummingbird itself and eliminates other noise.

## 6 CONCLUSION

We investigated the application of large-scale visionlanguage pre-trained models in continual learning. To capture semantic similarity, we employed text embeddings from the text encoder to compute category similarity. This information was then used to generate semantically-guided label supervision, enhancing the model's understanding of category relationships during training. Additionally, we proposed a refinement technique that improved distillation loss computation by considering the semantic similarity between text embeddings of old and new classes. This approach facilitated a more precise transfer of knowledge from previous tasks to new ones.

Limitations. Our method relies on the availability of text information for each task or category. In some real-world scenarios, such as image-only datasets or domains where textual descriptions are not readily available, our approach may not be directly applicable, which presents a potential avenue for future exploration.

## REFERENCES

- [1] F. M. Castro, M. J. Mar´ ın-Jim´ enez, N. Guil, C. Schmid, and K. Alahari, 'End-to-end incremental learning,' in Proceedings of the European conference on computer vision (ECCV) , pp. 233-248, 2018.
- [2] S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin, 'Learning a unified classifier incrementally via rebalancing,' in Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , pp. 831-839, 2019.
- [3] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, 'icarl: Incremental classifier and representation learning,' in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pp. 2001-2010, 2017.
- [4] B. Zhao, X. Xiao, G. Gan, B. Zhang, and S.-T. Xia, 'Maintaining discrimination and fairness in class incremental learning,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 13208-13217, 2020.
- [5] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. , 'Overcoming catastrophic forgetting in neural networks,' Proceedings of the national academy of sciences , vol. 114, no. 13, pp. 3521-3526, 2017.
- [6] Z. Li and D. Hoiem, 'Learning without forgetting,' IEEE transactions on pattern analysis and machine intelligence , vol. 40, no. 12, pp. 2935-2947, 2017.
- [7] Z. Zhao, Z. Zhang, X. Tan, J. Liu, Y. Qu, Y. Xie, and L. Ma, 'Rethinking gradient projection continual learning: Stability/plasticity feature space decoupling,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 3718-3727, 2023.
- [8] A. Douillard, M. Cord, C. Ollion, T. Robert, and E. Valle, 'Podnet: Pooled outputs distillation for small-tasks incremental learning,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings , pp. 86-102, Springer, 2020.
- [9] A. Douillard, A. Ram´ e, G. Couairon, and M. Cord, 'Dytox: Transformers for continual learning with dynamic token expansion,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 9285-9295, 2022.
- [10] J. Serra, D. Suris, M. Miron, and A. Karatzoglou, 'Overcoming catastrophic forgetting with hard attention to the task,' in International Conference on Machine Learning , pp. 4548-4557, PMLR, 2018.
- [11] S. Yan, J. Xie, and X. He, 'Der: Dynamically expandable representation for class incremental learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 3014-3023, 2021.
- [12] D. Madaan, H. Yin, W. Byeon, J. Kautz, and P. Molchanov, 'Heterogeneous continual learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1598515995, 2023.
- [13] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. , 'An image is worth 16x16 words: Transformers for image recognition at scale,' in International Conference on Learning Representations , 2020.
- [14] Z. Wang, Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister, 'Learning to prompt for continual learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 139-149, 2022.
- [15] Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y. Lee, X. Ren, G. Su, V. Perot, J. Dy, et al. , 'Dualprompt: Complementary prompting for rehearsal-free continual learning,' in European Conference on Computer Vision , pp. 631-648, Springer, 2022.
- [16] Y.-M. Tang, Y.-X. Peng, and W.-S. Zheng, 'When prompt-based incremental learning does not meet strong pretraining,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 1706-1716, 2023.
- [17] G. Zhang, L. Wang, G. Kang, L. Chen, and Y. Wei, 'Slca: Slow learner with classifier alignment for continual learning on a pretrained model,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023.
- [18] L. Zhou, H. Palangi, L. Zhang, H. Hu, J. Corso, and J. Gao, 'Unified vision-language pre-training for image captioning and vqa,' in Proceedings of the AAAI conference on artificial intelligence , vol. 34, pp. 13041-13049, 2020.
- [19] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. , 'Learning transferable visual models from natural language supervision,' in

- International conference on machine learning , pp. 8748-8763, PMLR, 2021.
- [20] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, 'Uniter: Universal image-text representation learning,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings , pp. 104-120, Springer, 2020.
- [21] J.-T. Zhai, X. Liu, L. Yu, and M.-M. Cheng, 'Fine-grained knowledge selection and restoration for non-exemplar class incremental learning,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, pp. 6971-6978, 2024.
- [22] V. Thengane, S. Khan, M. Hayat, and F. Khan, 'Clip model is an efficient continual learner,' arXiv preprint arXiv:2210.03114 , 2022.
- [23] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, 'Learning to prompt for vision-language models,' International Journal of Computer Vision , vol. 130, no. 9, pp. 2337-2348, 2022.
- [24] B. E. Shepp and S. Ballesteros, Object perception: Structure and process . Psychology Press, 2013.
- [25] L. Yu, B. Twardowski, X. Liu, L. Herranz, K. Wang, Y. Cheng, S. Jui, and J. v. d. Weijer, 'Semantic drift compensation for classincremental learning,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 6982-6991, 2020.
- [26] J.-T. Zhai, X. Liu, A. D. Bagdanov, K. Li, and M.-M. Cheng, 'Masked autoencoders are efficient class incremental learners,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 19104-19113, 2023.
- [27] B. Lester, R. Al-Rfou, and N. Constant, 'The power of scale for parameter-efficient prompt tuning,' in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , Association for Computational Linguistics, 2021.
- [28] M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, 'Visual prompt tuning,' in European Conference on Computer Vision , pp. 709-727, Springer, 2022.
- [29] X. L. Li and P. Liang, 'Prefix-tuning: Optimizing continuous prompts for generation,' in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 4582-4597, 2021.
- [30] J. Pfeiffer, A. Kamath, A. R¨ uckl´, e K. Cho, and I. Gurevych, 'Adapterfusion: Non-destructive task composition for transfer learning,' in 16th Conference of the European Chapter of the Associationfor Computational Linguistics, EACL 2021 , pp. 487-503, Association for Computational Linguistics (ACL), 2021.
- [31] R. Wang, D. Tang, N. Duan, Z. Wei, X.-J. Huang, J. Ji, G. Cao, D. Jiang, and M. Zhou, 'K-adapter: Infusing knowledge into pretrained models with adapters,' in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pp. 1405-1418, 2021.
- [32] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. , 'Lora: Low-rank adaptation of large language models,' in International Conference on Learning Representations .
- [33] J. S. Smith, L. Karlinsky, V. Gutta, P. Cascante-Bonilla, D. Kim, A. Arbelle, R. Panda, R. Feris, and Z. Kira, 'Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 11909-11919, 2023.
- [34] M. D. McDonnell, D. Gong, A. Parvaneh, E. Abbasnejad, and A. van den Hengel, 'Ranpac: Random projections and pre-trained models for continual learning,' Advances in Neural Information Processing Systems , vol. 36, 2024.
- [35] D.-W. Zhou, H.-L. Sun, H.-J. Ye, and D.-C. Zhan, 'Expandable subspace ensemble for pre-trained model-based class-incremental learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 23554-23564, 2024.
- [36] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig, 'Scaling up visual and visionlanguage representation learning with noisy text supervision,' in International Conference on Machine Learning , pp. 4904-4916, PMLR, 2021.
- [37] Y. Lu, J. Liu, Y. Zhang, Y. Liu, and X. Tian, 'Prompt distribution learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 5206-5215, 2022.
- [38] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, 'Conditional prompt learning for vision-language models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1681616825, 2022.
- [39] M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan, 'Maple: Multi-modal prompt learning,' in Proceedings of the
- IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 19113-19122, 2023.
- [40] M. U. Khattak and etc., 'Self-regulating prompts: Foundational model adaptation without forgetting,' in ICCV , 2023.
- [41] P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao, 'Clip-adapter: Better vision-language models with feature adapters,' International Journal of Computer Vision , vol. 132, no. 2, pp. 581-595, 2024.
- [42] A. Krizhevsky, G. Hinton, et al. , 'Learning multiple layers of features from tiny images,' 2009.
- [43] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'Imagenet: A large-scale hierarchical image database,' in 2009 IEEE conference on computer vision and pattern recognition , pp. 248-255, Ieee, 2009.
- [44] L. Bossard, M. Guillaumin, and L. Van Gool, 'Food-101-mining discriminative components with random forests,' in Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 , pp. 446-461, Springer, 2014.
- [45] J. Krause, M. Stark, J. Deng, and L. Fei-Fei, '3d object representations for fine-grained categorization,' in Proc. IEEE Int. Conf. Comput. Vision Workshops , 2013.
- [46] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, 'Fine-grained visual classification of aircraft,' arXiv preprint arXiv:1306.5151 , 2013.
- [47] L. F. A. M. R. M. P. P., 'Caltech101,' tech. rep., California Institute of Technology, 2003.
- [48] M.-E. Nilsback and A. Zisserman, 'Automated flower classification over a large number of classes,' in 2008 Sixth Indian Conference on Computer Vision, Graphics &amp; Image Processing , pp. 722-729, IEEE, 2008.
- [49] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, 'Cub,' Tech. Rep. CNS-TR-2011-001, California Institute of Technology, 2011.
- [50] A. Khosla, N. Jayadevaprakash, B. Yao, and F.-F. Li, 'Novel dataset for fine-grained image categorization: Stanford dogs,' in Proc. CVPR workshop on fine-grained visual categorization (FGVC) , vol. 2, Citeseer, 2011.
- [51] Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu, 'Large scale incremental learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 374-382, 2019.
- [52] C. Zhang, N. Song, G. Lin, Y. Zheng, P. Pan, and Y. Xu, 'Fewshot incremental learning with continually evolved classifiers,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 12455-12464, 2021.
- [53] D.-W. Zhou, F.-Y. Wang, H.-J. Ye, L. Ma, S. Pu, and D.-C. Zhan, 'Forward compatible few-shot class-incremental learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 9046-9056, 2022.
- [54] G. Hinton, O. Vinyals, and J. Dean, 'Distilling the knowledge in a neural network,' in NIPS Deep Learning and Representation Learning Workshop , 2015.
- [55] C. Huang, Y. Li, C. C. Loy, and X. Tang, 'Learning deep representation for imbalanced classification,' in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 5375-5384, 2016.
- [56] M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, and J. van de Weijer, 'Class-incremental learning: survey and performance evaluation on image classification,' IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.
- [57] L. Van der Maaten and G. Hinton, 'Visualizing data using t-sne.,' Journal of machine learning research , vol. 9, no. 11, 2008.
- [58] H. Chefer, S. Gur, and L. Wolf, 'Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 397-406, 2021.
Lu Yu Zhe Tao Hantao Yao Joost Van de Weijer Changsheng Xu
Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning
Deep neural networks (DNNs) excel on fixed datasets but struggle with
incremental and shifting data in real-world scenarios. Continual learning
addresses this challenge by allowing models to learn from new data while
retaining previously learned knowledge. Existing methods mainly rely on visual
features, often neglecting the rich semantic information encoded in text. The
semantic knowledge available in the label information of the images, offers
important semantic information that can be related with previously acquired
knowledge of semantic classes. Consequently, effectively leveraging this
information throughout continual learning is expected to be beneficial. To
address this, we propose integrating semantic guidance within and across tasks
by capturing semantic similarity using text embeddings. We start from a
pre-trained CLIP model, employ the \emph{Semantically-guided Representation
Learning (SG-RL)} module for a soft-assignment towards all current task
classes, and use the Semantically-guided Knowledge Distillation (SG-KD) module
for enhanced knowledge transfer. Experimental results demonstrate the
superiority of our method on general and fine-grained datasets. Our code can be
found in
https://github.com/aprilsveryown/semantically-guided-continual-learning.