summaries: In this paper, the problem of forward knowledge transfer in continual learning settings is explored. The idea is to measure correlations between the learned tasks based on the notion of "trust region" which helps to identify the most similar learned tasks to the current task. The core idea is that frozen weights for similar past tasks can be relaxed to reuse them to learn the current task. Since task similarities are used for this purpose, this will not lead to catastrophic forgetting and at the same time helps to transfer knowledge. Experiments on four benchmarks are provided to demonstrate that the method is effective. **Summary:** The paper focuses on gradient projection (GP) for incremental learning. The authors motivate their work by stating that while approaches based on GP lead to superior performance in overcoming catastrophic forgetting, they suffer from a significant drawback. In GP, once one calculates the subspaces spanned by layerwise inputs for a task, say Task A, the network is then forced to only update the weights in an orthogonal direction to these subspaces for learning a subsequent task, say Task B (hence keeping important parameters for Task A intact and overcome catastrophic forgetting). However, suppose Task B and Task A are similar (an extreme case is when Task B is the continuation of Task A!). In that case, we know that the essential parameters for Task A are likely to be important for Task B and that the network could benefit by continuing to update the important parameters for Task A, which is not allowed in GP algorithms. This behavior has two consequences: 1) intransigence, i.e., the network won't be able to learn Task B as effectively as possible, and 2) the network will have reduced backward transfer. The backward transfer issue is apparent in the extreme case when Task B is a continuation of Task A, and the network's performance on Task A would have improved if it was able to learn Task B using the important weights for Task A!  The paper addresses this issue, with a simple, yet practical, solution. For a new task, the authors calculate the correlation between the subspaces calculated for old tasks and the new task (layerwise), keep track of the most correlated previous tasks for each layer, and denote them as (layerwise) Trust Regions. Next, the authors propose a scaled weight projection that allows for unfreezing the important parameters in the Trust Region of the new task, while learning this task, and learn the corresponding weights as part of the optimization process. Finally, the authors report results on Permuted MNIST (PMNIST), CIFAR100 Split, CIFAR100 Sup, and a sequence of 5-Datasets with 10-class classification which includes CIFAR-10, MNIST, SVHN, not-MNIST, and Fashion MNIST, in comparison with GP methods, regularization-based methods, and memory replay method. They show consistent improvement in accuracy, and more interestingly in backward transfer.   Some existing methods put restrictive constrains on the optimization space of the new task to prevent catastrophic forgetting, which may lead to unsatisfactory performance for the new tasks. This paper aims to facilitate the forward knowledge transfer based on an efficient characterization of task correlation. Main contributions can be summarized as follows: 1.Introduce a novel notion of ‘trust region’ based on the norm of gradient projection onto the subspace spanned by task inputs to measure task correlation. 2.Proposed a novel approach for the new task to leverage the knowledge of the strongly correlated old tasks in the Trust Region through a scaled weight projection. 3.Developed a continual learning approach, trust region gradient projection(TRGP) based on the introduced Trust Region, scaled weight projection and a module to construct task input subspace. 4.Compared to related state-of-the-art approaches, TRGP achieves substantial performance improvement on all Benchmarks. This paper proposes a continual learning method based on gradient projection memory (GPM) of Saha et al., which projects the gradient of each layer to be orthogonal to the input subspace of previous tasks. Motivated by the fact that the orthogonal projection can harm the performance by being too restrictive, the authors propose a heuristic algorithm to reduce the restriction. Specifically, the authors choose a subset of most "correlated" tasks and let the model change along the subspace of the correlated tasks.
abstractText: Catastrophic forgetting is one of the major challenges in continual learning. To address this issue, some existing methods put restrictive constraints on the optimization space of the new task for minimizing the interference to old tasks. However, this may lead to unsatisfactory performance for the new task, especially when the new task is strongly correlated with old tasks. To tackle this challenge, we propose Trust Region Gradient Projection (TRGP) for continual learning to facilitate the forward knowledge transfer based on an efficient characterization of task correlation. Particularly, we introduce a notion of ‘trust region’ to select the most related old tasks for the new task in a layer-wise and single-shot manner, using the norm of gradient projection onto the subspace spanned by task inputs. Then, a scaled weight projection is proposed to cleverly reuse the frozen weights of the selected old tasks in the trust region through a layer-wise scaling matrix. By jointly optimizing the scaling matrices and the model, where the model is updated along the directions orthogonal to the subspaces of old tasks, TRGP can effectively prompt knowledge transfer without forgetting. Extensive experiments show that our approach achieves significant improvement over related state-of-the-art methods.
references.authors: Rahaf Aljundi Francesca Babiloni Mohamed Elhoseiny Marcus Rohrbach Tinne Tuytelaars
references.title: Memory aware synapses: Learning what (not) to forget
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Yaroslav Bulatov
references.title: Notmnist dataset. Google (Books/OCR)
references.venue: Tech. Rep.[Online]. Available: http://yaroslavvb. blogspot. it/2011/09/notmnist-dataset. html,
references.authors: Arslan Chaudhry Puneet K Dokania Thalaiyasingam Ajanthan Philip HS Torr
references.title: Riemannian walk for incremental learning: Understanding forgetting and intransigence
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Arslan Chaudhry Marc’Aurelio Ranzato Marcus Rohrbach Mohamed Elhoseiny
references.title: Efficient lifelong learning with a-gem
references.venue: arXiv preprint arXiv:1812.00420,
references.authors: Arslan Chaudhry Marcus Rohrbach Mohamed Elhoseiny Thalaiyasingam Ajanthan Puneet K Dokania Philip HS Torr M Ranzato
references.title: Continual learning with tiny episodic memories. 2019
references.authors: Zhiyuan Chen Bing Liu
references.title: Lifelong machine learning
references.venue: Synthesis Lectures on Artificial Intelligence and Machine Learning,
references.authors: Prithviraj Dhar Rajat Vikram Singh Kuan-Chuan Peng Ziyan Wu Rama Chellappa
references.title: Learning without memorizing
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Mehrdad Farajtabar Navid Azizan Alex Mott Ang Li
references.title: Orthogonal gradient descent for continual learning
references.venue: In International Conference on Artificial Intelligence and Statistics,
references.authors: Yunhui Guo Mingrui Liu Tianbao Yang Tajana Rosing
references.title: Improved schemes for episodic memory based lifelong learning algorithm
references.venue: In Conference on Neural Information Processing Systems,
references.authors: Steven CY Hung Cheng-Hao Tu Cheng-En Wu Chien-Hung Chen Yi-Ming Chan Chu-Song Chen
references.title: Compacting, picking and growing for unforgetting continual learning
references.authors: James Kirkpatrick Razvan Pascanu Neil Rabinowitz Joel Veness Guillaume Desjardins Andrei A Rusu Kieran Milan John Quan Tiago Ramalho Agnieszka Grabska-Barwinska
references.title: Overcoming catastrophic forgetting in neural networks
references.venue: Proceedings of the national academy of sciences,
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning multiple layers of features from tiny images
references.authors: Sang-Woo Lee Jin-Hwa Kim Jaehyun Jun Jung-Woo Ha Byoung-Tak Zhang
references.title: Overcoming catastrophic forgetting by incremental moment matching
references.venue: arXiv preprint arXiv:1703.08475,
references.authors: Xilai Li Yingbo Zhou Tianfu Wu Richard Socher Caiming Xiong
references.title: Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting
references.venue: In International Conference on Machine Learning,
references.authors: Zhizhong Li Derek Hoiem
references.title: Learning without forgetting
references.venue: IEEE transactions on pattern analysis and machine intelligence,
references.authors: David Lopez-Paz Marc’Aurelio Ranzato
references.title: Gradient episodic memory for continual learning
references.venue: Advances in neural information processing systems,
references.authors: Michael McCloskey Neal J Cohen
references.title: Catastrophic interference in connectionist networks: The sequential learning problem
references.venue: In Psychology of learning and motivation,
references.authors: Yuval Netzer Tao Wang Adam Coates Alessandro Bissacco Bo Wu Andrew Y Ng
references.title: Reading digits in natural images with unsupervised feature learning
references.authors: Matthew Riemer Ignacio Cases Robert Ajemian Miao Liu Irina Rish Yuhai Tu Gerald Tesauro
references.title: Learning to learn without forgetting by maximizing transfer and minimizing interference
references.authors: Hippolyt Ritter Aleksandar Botev David Barber
references.title: Online structured laplace approximations for overcoming catastrophic forgetting
references.venue: arXiv preprint arXiv:1805.07810,
references.authors: Amir Rosenfeld John K Tsotsos
references.title: Incremental learning through deep adaptation
references.venue: IEEE transactions on pattern analysis and machine intelligence,
references.authors: Gobinda Saha Isha Garg Kaushik Roy
references.title: Gradient projection memory for continual learning
references.venue: arXiv preprint arXiv:2103.09762,
references.authors: John Schulman Sergey Levine Pieter Abbeel Michael Jordan Philipp Moritz
references.title: Trust region policy optimization
references.venue: In International conference on machine learning,
references.authors: Jonathan Schwarz Wojciech Czarnecki Jelena Luketina Agnieszka Grabska-Barwinska Yee Whye Teh Razvan Pascanu Raia Hadsell
references.title: Progress & compress: A scalable framework for continual learning
references.venue: In International Conference on Machine Learning,
references.authors: Joan Serra Didac Suris Marius Miron Alexandros Karatzoglou
references.title: Overcoming catastrophic forgetting with hard attention to the task
references.venue: In International Conference on Machine Learning,
references.authors: Hanul Shin Jung Kwon Lee Jaehong Kim Jiwon Kim
references.title: Continual learning with deep generative replay
references.venue: arXiv preprint arXiv:1705.08690,
references.authors: Tom Veniat Ludovic Denoyer Marc’Aurelio Ranzato
references.title: Efficient continual learning with modular networks and task-driven priors
references.venue: arXiv preprint arXiv:2012.12631,
references.authors: Oriol Vinyals Charles Blundell Timothy Lillicrap Daan Wierstra
references.title: Matching networks for one shot learning
references.venue: Advances in neural information processing systems,
references.authors: Han Xiao Kashif Rasul Roland Vollgraf
references.title: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms
references.venue: arXiv preprint arXiv:1708.07747,
references.authors: Ju Xu Zhanxing Zhu
references.title: Reinforced continual learning
references.venue: arXiv preprint arXiv:1805.12369,
references.authors: Jaehong Yoon Eunho Yang Jeongtae Lee Sung Ju Hwang
references.title: Lifelong learning with dynamically expandable networks
references.venue: arXiv preprint arXiv:1708.01547,
references.authors: Jaehong Yoon Saehoon Kim Eunho Yang Sung Ju Hwang
references.title: Scalable and order-robust continual learning with additive parameter decomposition
references.venue: In Eighth International Conference on Learning Representations,
references.authors: Guanxiong Zeng Yang Chen Bo Cui Shan Yu
references.title: Continual learning of context-dependent processing in neural networks
references.venue: Nature Machine Intelligence,
references.authors: Friedemann Zenke Ben Poole Surya Ganguli
references.title: Continual learning through synaptic intelligence
references.venue: In International Conference on Machine Learning,
sections.heading: 1 INTRODUCTION
sections.text: Human beings can continuously learn different new tasks without forgetting the learnt knowledge of old tasks in their lifespan. Aiming to achieve this remarkable capability for the deep neural networks (DNNs), continual learning (CL) (Chen & Liu, 2018) has garnered much attention in recent years. Nevertheless, many existing CL methods still leave the DNN vulnerable to forget the knowledge of old tasks when learning new tasks. Such a phenomenon is known as ‘Catastrophic Forgetting’ (McCloskey & Cohen, 1989), which has become one of the major challenges for CL.
  Many approaches (e.g., (Rusu et al., 2016; Li & Hoiem, 2017; Dhar et al., 2019; Guo et al., 2020; Zeng et al., 2019)) have been proposed to address the forgetting issue, which can be generally divided into two classes depending on the network architecture, i.e., expansion methods and nonexpansion methods. In order to understand the fundamental limit of a fixed capacity neural network, we focus on non-expansion methods in this work. The basic idea for non-expansion methods is to constrain the gradient update either explicitly or implicitly when learning the new task, so as to minimize the introduced interference to old tasks. For example, the regularization-based methods (e.g., (Kirkpatrick et al., 2017; Serra et al., 2018)) penalize the modification on the most important weights of old tasks through model regularizations; experience-replay based methods (e.g., (Shin et al., 2017; Chaudhry et al., 2019)) constrain the gradient directions by replaying the data of old tasks during learning of new tasks, in the format of either real data or synthetic data from generative models; and orthogonal-projection based methods (e.g., (Farajtabar et al., 2020; Saha et al., 2021)) update the model with gradients in the orthogonal directions of old tasks, without the access to old task data. In particular, the recently proposed Gradient Projection Memory (GPM) (Saha et al., 2021) has demonstrated superior performance compared to other approaches.
  To sufficiently minimize the interference to old tasks, most existing non-expansion methods (particularly the orthogonal-projection based methods), often put restrictive constraints on the optimization space of the new task, which may throttle the learning performance for the new task. A plausible conjecture is that such a scenario is likely to occur when the new task is strongly correlated with old tasks, and in this study we provide evidence to support this conjecture. The underlying rationale
  is as follows: The weights that are important to the new task are also important to the old tasks strongly correlated with the new task, which are often frozen to address the forgetting in the existing methods; however, they should be updated in the learning of the new task.
  To tackle this challenge, a key insight is that for a new task that is strongly correlated with old tasks, although the model optimization space could be more restrictive, there should be better forward knowledge transfer from the correlated old tasks to the new task. With this insight, we propose an innovate continual learning approach to facilitate the forward knowledge transfer without forgetting. The main contributions can be summarized as follows:
  (1) Inspired by (Schulman et al., 2015), we introduce a novel notion of ‘trust region’ based on the norm of gradient projection onto the subspace spanned by task inputs, which selects the old tasks strongly correlated to the new task in a layer-wise and single-shot manner. Intuitively, the new task and the selected old tasks in the trust region have similar input features for the corresponding layer.
  (2) We propose a novel approach for the new task to leverage the knowledge of the strongly correlated old tasks in the trust region through a scaled weight projection. Particularly, a scaling matrix is learnt in each layer for the new task to scale the weight projection onto the subspace of old tasks in the trust region, in order to reuse the frozen weights of old tasks without modifying the model.
  (3) Building on the introduced trust region, scaled weight projection, and a module to construct task input subspace, we develop a continual learning approach, trust region gradient projection (TRGP), that jointly optimizes the scaling matrices and the model for the new task. To mitigate the forgetting issue further, the model is updated along the directions orthogonal to the subspaces of old tasks.
  (4) We evaluate TRGP on standard CL benchmarks using various network architectures. Compared to related state-of-the-art approaches, TRGP achieves substantial performance improvement on all benchmarks, and demonstrates universal improvement on all tasks. The superior performance indicates that TRGP can effectively promote the forward knowledge transfer while alleviating forgetting.
sections.heading: 2 RELATED WORK
sections.text: Expansion-based methods. Expansion-based methods (e.g., (Rusu et al., 2016; Li & Hoiem, 2017; Rosenfeld & Tsotsos, 2018; Hung et al., 2019; Yoon et al., 2017; Li et al., 2019; Veniat et al., 2020)) dynamically expand the network capacity to reduce the interference between the new tasks and the old ones. Progressive Neural Network (PNN) (Rusu et al., 2016) expands the network architecture for new tasks and preserves the weights of old tasks. Learning Without Forgetting (LWF) (Li & Hoiem, 2017) splits the model layers into two parts, i.e., the shared part co-used by all tasks, and the task-specific part which grows for new tasks. Dynamic-Expansion Net (DEN) (Yoon et al., 2017) and Compacting-Picking-Growing (CPG) (Hung et al., 2019) combine the strategies of model compression/pruning, weight selection and model expansion. In order to find the optimal structure for each of the sequential tasks, Reinforced Continual Learning (RCL) (Xu & Zhu, 2018) leverages reinforcement learning and (Li et al., 2019) adapts architecture search. APD (Yoon et al., 2020) adds additional task-specific parameters for each task and selectively learns the task-shared parameters.
  Regularization-based methods. This category of methods (e.g., (Kirkpatrick et al., 2017; Lee et al., 2017; Chaudhry et al., 2018a; Dhar et al., 2019; Ritter et al., 2018; Schwarz et al., 2018; Zenke et al., 2017)) protect the old tasks by adding regularization terms in the loss function to penalize the model change on their important weights. Notably, to determine the weight importance, Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) leverages Fisher information matrix, HAT (Serra et al., 2018) learns hard attention masks. MAS (Aljundi et al., 2018) evaluates the model outputs sensitivity to the inputs in an unsupervised manner.
  Memory-based methods. Depending on if data of old tasks is utilized when learning new tasks, memory-based methods can be further divided into the following two categories. 1) Experiencereplay based methods. This class of methods replays the old tasks data along with the current task data to mitigate catastrophic forgetting. Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017) and Averaged GEM (A-GEM) (Chaudhry et al., 2018b) alter the current gradient based on the gradient computed with data in the memory. A unified view of episodic memory based methods is proposed in (Guo et al., 2020), based on new approaches are developed to balance between old tasks and the new task. Tiny episodic memory is considered in (Chaudhry et al., 2019) and meta-
  learning is leveraged in (Riemer et al., 2018). 2) Orthogonal-projection based method. To eliminate the need of storing data of old tasks, recently a series work (Zeng et al., 2019; Farajtabar et al., 2020; Saha et al., 2021) updates the model in the orthogonal direction of old tasks, and has shown remarkable performance. Particularly, Orthogonal Weight Modulation (OWM) (Zeng et al., 2019) learns a projector matrix to multiply with the new gradients. Orthogonal Gradient Descent (OGD) (Farajtabar et al., 2020) stores the gradient directions of old tasks and projects the new gradients on the directions orthogonal to the subspace spanned by the old gradients. Gradient Projection Memory (GPM) (Saha et al., 2021) stores the bases of the subspaces spanned by old task data and projects the new gradients on the directions orthogonal to these subspaces.
sections.heading: 3 PROBLEM FORMULATION
sections.text: Continual learning. Consider the setting where a sequence of tasks T = {t}Tt=1 arrives sequentially. Each task t has a dataset Dt = {(xt,i,yt,i)}Nti=1 with Nt sample pairs, where xt,i is the input vector and yt,i is the label vector. Consider a fixed capacity neural network with L layers, and the set of weights is denoted as W = {W l}Ll=1, whereW l is the layer-wise weight for layer l. Given the data input xt,i for task t, denote xlt,i as the input of layer l and x 1 t,i = xt,i. The output x l+1 t,i for layer l is computed as xl+1t,i = f(W l,xlt,i), where f is the operation of the network layer. Following (Saha et al., 2021), we denote xlt,i as the representations of xt,i at layer l. When learning a task t, we only have access to dataset Dt. Let L(W, {(xt,i,yt,i)}) = Lt(W) denote the loss function for training, e.g., mean squared and cross-entropy loss, and Wt denote the model after learning task t.
  Orthogonal-projection based methods. To minimize the interference to old tasks, recently a series of studies (Zeng et al., 2019; Farajtabar et al., 2020; Saha et al., 2021) has been carried out to update the model for the new task in the direction orthogonal to the subspace spanned by inputs of old tasks. In what follows, we briefly introduce the main ideas through a basic case with two tasks 1 and 2.
  Denote the subspace spanned by the inputs of task 1 for layer l as Sl1 and the learnt model for task 1 as W1 = {W l1}Ll=1. It is clear that xl1,i ∈ Sl1. When learning task 2, the modelW l1 will be modified in the direction orthogonal to Sl1, by either multiplying the gradient∇W lL2 with a projector matrix (e.g, (Zeng et al., 2019)), or projecting the gradient∇W lL2 onto the orthogonal direction to Sl1 (e.g., (Saha et al., 2021)). Let ∆W l1 denote the model change after learning task 2. It follows immediately that ∆W l1x l 1,i = 0, and the modelW l 2 for task 2 isW l 2 = W l 1 + ∆W l 1. Therefore, for task 1:
  W l2x l 1,i = (W l 1 + ∆W l 1)x l 1,i = W l 1x l 1,i + ∆W l 1x l 1,i = W l 1x l 1,i, (1)
  which indicates that no interference is introduced to task 1 after learning task 2, thereby addressing the forgetting issue. Such an analysis can be generalized to a sequence of tasks.
  When would orthogonal projection hinder the learning of a new task? Orthogonal projection provides a promising solution to address the forgetting in continual learning. However, by modifying the model only in the orthogonal direction to the input space of old tasks, the optimization space of learning the new task could be more restrictive, resulting in compromised performance of the new task. To get a more concrete sense, consider the following basic examples with two tasks 1 and 2.
  (Toy example 1) Suppose task 1 has dataset D1 = {(xi,yi)}Ni=1 and task 2 has dataset D2 = {(−xi,yi)}Ni=1, where only the sign is changed for the input vectors. Consider the case where two tasks share the same classifier (Saha et al., 2021). It is clear that for the l-th layer, the subspace spanned by {xli}Ni=1 of task 1 is same with the subspace spanned by {−xli}Ni=1 of task 2, i.e., Sl1 = S l 2, given the learnt model W l 1 for task 1. Based on the fact that stochastic gradient descent updates lie in the subspace spanned by the data input (Zhang et al., 2021; Saha et al., 2021), it follows that the gradient ∇W lL2 ∈ Sl2, such that ∇W lL2 ∈ Sl1. Therefore, the projection of ∇W lL2 onto the orthogonal direction to Sl1 is 0, which means that the model W l 1 will not be updated when learning task 2, i.e., W l2 = W l 1. However, the optimal model for task 2 should be W l 2 = −W l1, becauseW l1x l i achieves the minimum loss for the label yi after learning task 1.
  (Toy example 2) Suppose the input subspace of task 1 is orthogonal to that of task 2, i.e., Sl1 ⊥ Sl2. It follows that the projection of∇W lL2 onto the orthogonal direction to Sl1 is indeed equal to∇W lL2. Consequently, updating the model for task 2 based on orthogonal projection will not only introduce no interference to task 1, but also move along the direction of steepest descent for task 2.
  Motivated by these examples, a plausible conjecture is that naive orthogonal projection could possibly compromise the learning performance of the new task that is strongly correlated with old tasks, especially when the correlation is “negative” as in the toy example 1. In this study, we advocate to characterize the task correlation through the correlation between the input subspaces for two tasks. As illustrated in Fig. 1, when the subspace is 2-dimensional, two tasks are weakly correlated if their input subspaces are nearly orthogonal, and strongly correlated if their subspaces are nearly parallel.
sections.heading: 4 TRUST REGION GRADIENT PROJECTION FOR CONTINUAL LEARNING
sections.text: To tackle these challenges, a key insight is that for a new task that is strongly correlated with old tasks, although the model optimization space could be more restrictive, there should be better forward knowledge transfer from the correlated old tasks to the new task. With this insight, we propose a novel approach to prompt forward knowledge transfer without forgetting, by 1) introducing a novel notion of trust region to select the most related old tasks in a single-shot manner and 2) cleverly reusing the frozen weights of the selected tasks in the trust region with a scaled weight projection.
sections.heading: 4.1 TRUST REGION
sections.text: To facilitate forward knowledge transfer from the correlated old tasks to the new task, the first question is how to efficiently select the most correlated old tasks. Towards this end, we characterize the correlation between the input subspaces for two tasks, through the lens of gradient projection.
  Specifically, denote Slj = span{Blj} as the subspace spanned by the task j data for layer l, where Blj = [u l j,1, ...,u l j,Mj,l
  ] is the bases for Slj (totally Mj,l bases extracted from the input). For any matrixA with a suitable dimension, denote its projection onto the subspace Slj as:
  ProjSlj (A) = ABlj(B l j) ′ (2)
  where (·)′ is the matrix transpose. We next define a layer-wise trust region for a new task as a set of its most related old tasks, based on the norm of projected gradient onto the subspaces of old tasks. Definition 1 (Layer-Wise Trust Region). For any new task t ≥ 2 and layer l, we define a layer-wise trust region T Rlt = {j} for j ∈ [1, t− 1], where for any task j ∈ T R l t the following holds:
  ‖ProjSlj (∇W lLt(Wt−1))‖2 ≥ l‖∇W lLt(Wt−1)‖2, (3)
  where l ∈ [0, 1] and Wt−1 is the model after learning task t− 1.
  Intuitively, for the new task t, the norm of its gradient projection onto the subspace of an old task j serves as a surrogate for characterizing the correlation between input subspaces for these two tasks, due to the fact that the gradient lies in the span of its input. When the condition Eq. (3) is satisfied, the gradient ∇W lLt(Wt−1) has a large projection onto the subspace of an old task j, which implies that the subspace Slt for task t and the subspace Slj for task j may have sufficient common bases for layer l. In this case, we trust that the old task
  j is strongly correlated with the new task t in layer l, and put it into task t’s trust region T Rlt. A simple illustration of trust region is shown in Figure 2. Note that the notion of trust region can also be generalized to a task-wise definition, where the most correlated old tasks will be selected based on the projection of the entire gradient ∇WLt(Wt−1). However, the layer-wise trust region could select different tasks for different layers, which provides a more fine-resolution characterization of task correlations in terms of layer-level features.
  Practical implementation. Besides the valuable functionality provided by the trust region for selecting most correlated old tasks, another significant benefit is the simplicity of its practical implementation. Consider the implementation for learning a new task t.
  (1) Single-shot manner. Given the learnt model Wt−1, we select a sample batch from dataset Dt, and compute the gradient∇W lLt(Wt−1) in one forward-backward pass for all layers at once. Given the subspace Slj for an old task j, the condition Eq. (3) can be immediately evaluated for all old tasks.
  (2) Top-K correlated tasks. It is clear that the choice of l has a nontrivial impact on the selection of the most correlated old tasks. To reduce the sensitivity of the performance on l, we can set a relatively small value of l, and pick the top-K old tasks with the largest gradient projection norm ‖ProjSlj (∇W lLt(Wt−1))‖2 from the tasks satisfying Eq. (3). As demonstrated later in our experiments, setting K = 1 is enough to achieve a significant performance improvement.
sections.heading: 4.2 SCALED WEIGHT PROJECTION
sections.text: Given the layer-wise trust region T Rlt for the new task t, the next key question is how to efficiently leverage the knowledge of the most correlated old tasks in T Rlt for learning task t. To this end, we propose a novel approach to reuse the frozen weights of the selected old tasks in T Rlt through a scaled weight projection with a scaling matrix.
  At the outset, it is of interest to understand what knowledge is preserved for old tasks during continual learning with orthogonal projection. Based on Eq. (1) for the simple case with two learning tasks 1 and 2 as mentioned earlier, it can be shown that
  ProjSl1 (W l2) = ProjSl1 (W l1 + ∆W l 1) = ProjSl1 (W l1) + ProjSl1 (∆W l1) = ProjSl1 (W l1) (4)
  where the last equation holds because the model W l1 is updated in the direction orthogonal to S l 1 when learning task 2. By generalizing Eq. (4) to the case with a sequence of tasks, we can have that for the model Wt−1 after learning task t− 1 and any old task j < t:
  ProjSlj (W lt−1) = ProjSlj (W lj ), (5)
  which indicates that the model weight projection on the subspace of old tasks is actually “frozen” during continual learning so as to overcome forgetting of the old tasks.
  On the other hand, because the trust region T Rlt is constructed in a way that the subspace Slt of task t is strongly correlated with the subspace Slj for any old task j ∈ T R l t, the bases B l j of Slj is very likely to contain important bases for task t. As a result, the weight projection ProjSlj (W l t−1) is important for the new task t and should be modified accordingly in order to guarantee the learning performance of task t, which however has to be frozen to protect task j. To find an efficient way to leverage ProjSlj (W l t−1) without modifying it, note that the projection ProjSlj (W l t−1) is indeed a linear combination of the projection of W lt−1 onto each basis in B l j , and every point in S l j can be obtained by scaling the coordinates of ProjSlj (W l t−1). Figure 3 shows a simple example for two-dimensional subspace. Therefore, we propose a scaled weight projection to find the best point for task t in Slj by leveraging the projection ProjSlj (W l t−1) through a square scaling matrixQ l j,t:
  ProjQ Slj (W lt−1) = W l t−1B l jQ l j,t(B l j) ′. (6)
  The dimension of Qlj,t depends on the number of bases in B l j (dimension of S l j), which is usually small for each task. In this way, we explicitly transfer the knowledge of the selected old tasks in the trust region T Rlt to the new task t through a scaling matrixQlj,t.
sections.heading: 4.3 TASK SUBSPACE CONSTRUCTION
sections.text: To successfully leverage the trust region, a missing ingredient is the construction of input subspaces of old tasks. We next show how the subspace Slj can be constructed for task j at layer l.
  For task j = 1. As in (Saha et al., 2021), we obtain the basesBl1 after learning task 1 using Singular Value Decomposition (SVD) on the representations. Specifically, given the model W1 after learning task 1, we construct a representation matrix Rl1 = [x l 1,1, ...,x l 1,n] ∈ Rm×n with n samples, where each xl1,i ∈ Rm, is the representation at layer l by forwarding the sample x1,i through the network. Then, we apply SVD to the matrix Rl1, i.e., R l 1 = U l 1Σ l 1(V l 1 ) ′, where U l1 = [u l 1,1, ...,u l 1,m] ∈ Rm×m is an orthogonal matrix with left singular vector ul1,i ∈ Rm, V l1 = [vl1,1, ...,vl1,n] ∈ Rn×n is an orthogonal matrix with right singular vector vl1,i ∈ Rn, and Σl1 ∈ Rm×n is a rectangular diagonal matrix with non-negative singular values {σl1,i} min{m,n} i=1 on the diagonal in a descending order. To obtain the bases for subspace Sl1, we use k l 1-rank matrix approximation to pick the first k l 1 left singular vectors inU l1, such that the following condition is satisfied for a threshold η l th ∈ (0, 1):
  ‖(Rl1)kl1‖ 2 F ≥ lth‖Rl1‖2F (7) where (Rl1)kl1 = ∑kl1 i=1 σ l 1,iu l 1,i(v l 1,i) ′ is a kl1-rank (k l 1 ≤ r) approximation of the representation matrix Rl1 with rank r ≤ min{m,n}, and ‖ · ‖F is the Frobenius norm. Then the bases Bl1 for subspace Sl1 can be constructed asB l 1 = [u l 1,1, ...,u
  l 1,kl1 ].
  For task j ∈ [2, T ]. We construct the bases Blj after learning task j given the learnt model Wj . A representation matrix Rlj will be first obtained in the same manner as R l 1. Note that the bases {Bli} j−1 i=1 learnt for old tasks may include important bases for task j. Therefore, we learn the bases Blj by selecting the most important bases from both bases of old tasks and newly constructed bases. Specifically, (1) (old bases) we first concatenate the bases {Bli} j−1 i=1 of old tasks together inM l j and eliminate the common bases. For each basis uli ∈M lj , we compute the corresponding eigenvalue of Rlj(R l j) ′, i.e., δli = (u l i) ′Rlj(R l j) ′uli, which is the square of the singular value ofR l j with respect to uli. (2) (new bases) We perform SVD on R̂lj = Rlj −RljM lj(M lj)′ to generate new bases beyond M lj , i.e., R̂ l j = Û l jΣ̂ l j(V̂ l j ) ′ with singular values {σ̂lj,h}h. (3) (select the most important bases from both old and new bases) Next we concatenate {δli}i and {(σ̂lj,h)2}h together in a vector δ, and sort them in a descending order. We perform klj-rank matrix approximation ofR l j , such that the summation of the first klj elements in δ is greater than l th‖Rlj‖2F . Then Blj can be constructed by selecting the bases corresponding to the first klj elements in δ.
sections.heading: 4.4 CONTINUAL LEARNING WITH TRUST REGION GRADIENT PROJECTION
sections.text: Building on the three modules proposed earlier, i.e., task subspace construction, trust region and scaled weight projection, we next present our approach TRGP for continual learning that efficiently facilitate forward knowledge transfer without forgetting the old tasks.
  Learning task 1. The first task is learnt using standard gradient descent. The subspace {Sl1}Ll=1 is constructed by following Section 4.3.
  Learning task 2, ..., T. For task t ∈ [2, T ], we first determine the trust region T Rlt with top-K correlated old tasks selected for layer l. The optimization problem for task t is as follows:
  min {W l}l,{Qlj,t}l,j∈TRlt
  L({W leff}l,Dt), (8)
  s.t W leff = W l + ∑ j∈TRlt [ProjQ Slj (W l)− ProjSlj (W l)], (9)
  where the gradient for updatingW l is∇W lL = ∇W lL− (∇W lL)M lt (M lt )′ andM lt is the bases of all old tasks as in Section 4.3. The subspace {Slt}Ll=1 is next obtained by following Section 4.3.
sections.heading: 5 EXPERIMENTAL RESULTS
sections.heading: 5.1 EXPERIMENTAL SETUP
sections.text: Datasets and training details. We evaluate our method on multiple datasets against state-of-the-art CL methods. 1) PMNIST. Following (Lopez-Paz & Ranzato, 2017; Saha et al., 2021), we create 10 sequential tasks using different permutations where each task has 10 classes. We use a 3-layer fully-connected network. 2) CIFAR-100 Split. We split the classes of CIFAR-100 (Krizhevsky et al., 2009) into 10 group, and consider 10-way multi-class classification in each group as a single task. Similar with (Serra et al., 2018; Saha et al., 2021), we use a version of 5-layer AlexNet. 3) CIFAR-100 Sup. We divide the CIFAR-100 dataset into 20 tasks where each task has 5 classes. We use a modified version of LeNet-5. 4) 5-Datasets. We use a sequence of 5-Datasets which includes CIFAR-10, MNIST, SVHN (Netzer et al., 2011), not-MNIST (Bulatov, 2011) and Fashion MNIST (Xiao et al., 2017), where each dataset is set to be a task. We adapt a reduced ResNet18 network that is used in (Lopez-Paz & Ranzato, 2017). 5) MiniImageNet Split. We split the 100 classes of MiniImageNet (Vinyals et al., 2016) into 20 sequential tasks where each task has 5 classes, and consider a reduced ResNet18 network. In addition, for all the experiments, the threshold l is set to 0.5, and we select top-2 tasks that satisfy condition Eq. (3). We use the same threshold lth as GPM (Saha et al., 2021) for subspace construction. More details are in the appendix.
  Methods for comparison. To test the efficacy of our method, we compare it with state-of-the-art approaches in three categories: 1) Memory-based methods. We compare with Experience Replay with reservoir sampling (ER Res) (Chaudhry et al., 2019), Averaged GEM (A-GEM) (Chaudhry et al., 2018b), Orthogonal Weight Modulation (OWM) (Zeng et al., 2019) and Gradient Projection Memory (GPM) (Saha et al., 2021). 2) Regularization-based methods. We compare with state-ofthe-art HAT (Serra et al., 2018) and Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017). 3) Expansion-based methods. We further compare with Progressive Neural Network (PNN) (Rusu et al., 2016), Learning Without Forgetting (LWF) (Li & Hoiem, 2017), Dynamic-Expansion Net (DEN) (Yoon et al., 2017), and APD (Yoon et al., 2020), by using CIFAR-100 Sup dataset.
  Metrics. Following GPM (Saha et al., 2021), two metrics are used to evaluate the performance: Accuracy (ACC), the average final accuracy over all tasks, and Backward Transfer (BWT), which measures the forgetting of old tasks when learning new tasks. ACC and BWT are defined as:
  ACC = 1
  T ∑T i=1 AT,i, BWT = 1 T − 1 ∑T−1 i=1 AT,i −Ai,i (10)
  where T is the number of tasks, AT,i is the accuracy of the model on i-th task after learning the T -th task sequentially.
sections.heading: 5.2 MAIN RESULTS
sections.text: ACC and BWT comparison. As shown in Table 1, TRGP achieves significantly accuracy improvement compared with prior works on all datasets. For example, in contrast to the best prior results, TRGP achieve the accuracy gain of 2.43%, 1.98% and 1.37% over GPM on PMNIST, CIFAR-100 Split and MiniImageNet, respectively, and 2.34% over HAT on 5-Dataset. Surprisingly, we could even achieve better accuracy than Multitask on 5-Datasets, which usually serves as an upper bound for CL benchmarks. This superior performance of TRGP clearly shows its capability to effectively facilitate forward knowledge transfer. In addition, TRGP also demonstrates strong performance with the lowest BWT, reducing 0.2% than OWM and 0.6% than GPM, even with 5.63% and 2.34% ac-
  curacy improvement on PMNIST and 5-Dataset, respectively. Compared with HAT on CIFAR-100 Split, TRGP has marginally worse BWT, but achieves 2.4% accuracy gain.
  Moreover, TRGP exhibits an universal dominance over GPM about the final accuracy of all tasks on all the three datasets. According to the apple to apple comparison with GPM in Fig. 4, one interesting phenomenon is observed: TRGP has the similar accuracy on “easy” tasks, but significantly improves the accuracy on the “difficult” tasks. For example, in the 5-Dataset setting, both TRGP and GPM achieve good accuracy on Task 1 (MNIST) and 3 (Fashion MNIST), which can be easily trained well, but TRGP significantly outperforms GPM on the rest three more difficult Tasks (CIFAR-10, SVHN and NotMNIST). In the end, as shown in Table 2, we further compare with the expansionbased methods by using CIFAR-100 Sup setting. It can be seen that TRGP outperforms all other CL methods, with a fixed capacity network.
  Discussion. We next show the accuracy evolution of specific tasks during the training of all tasks sequentially. We randomly select three tasks for each dataset to compare with GPM (we only show results on PMNIST in Fig. 5 and relegate the rest to the appendix). There are two main obervations: 1) TRGP completely outperforms GPM during training for all the sequential tasks on the three datasets; 2) For the PMNIST and 5-Dataset settings, TRGP could significantly reduce forgetting. To understand why, consider the case where GPM and TRGP learns a new task t given the same model Wt−1, and denote {M lt−1}l as the bases of all old tasks. Then we can have
  For GPM, the effective weight for layer l is W leff = ProjMlt−1 (W l) + Proj⊥Mlt−1 (W l) (11) where the weight projection on M lt−1 is frozen to protect old tasks, and only the weight projection orthogonal toM lt−1 can be updated for learning task t.
  For TRGP, the effective weight for layer l is W leff = Proj{Slj}j /∈TRlt (W l) + ProjQ{Slj}j∈TRlt (W l) + Proj⊥Mlt−1 (W l) (12) where only the first term, i.e., weight projection on subspaces of old tasks that are not in the trust region T Rlt, is frozen for task t. In contrast to GPM, an additional and also important part of weights, i.e., the scaled weight projection on subspaces of related old tasks in T Rlt, can be learnt in a favorable way for task t. As a result, TRGP can achieve better forward knowledge transfer by explicitly and cleverly reusing the important knowledge of strongly correlated old tasks in the trust region. More interestingly, benefiting from the task-unique information captured by the scaled weight projection, the backward transfer can also be reduced.
sections.heading: 5.3 ABLATION STUDY AND ANALYSIS
sections.text: Impact of the threshold l. To understand the impact of the threshold l, we evaluate the learning performance for three different values of l (i.e., 0.2, 0.5, 0.7) as shown in Table 3. The results show that the accuracy is very stable across the three threshold values, with ignoble accuracy difference on both CIFAR-100 Split and 5-Dataset settings. The reason behind is because we only select top-2 old tasks with largest gradient projection norm into the trust region, among all tasks satisfying condition Eq. (3). Therefore, for a wide range of l, the selected tasks in the trust region are actually fixed. The small accuracy fluctuation is because with some possibility only one old task satisfies Eq. (3) and is selected for some layers when l increases. Overall, TRGP is very robust to the value of l.
  Layer-wise vs. Task-wise trust region. To show the efficacy of layer-wise trust region, we compare it with the task-wise variant which shares a fixed trust region across all layers for each task. First, as shown in Table 3, layer-wise could achieve 1.21% accuracy gain over task-wise on CIFAR-100 Split. Furthermore, we illustrate the final accuracy of all tasks for layer-wise and taskwise of the proposed TRGP, and GPM on CIFAR-100 Split setting in Fig. 6. First, the performance of layer-wise is better than or comparable to task-wise for all tasks, because layer-wise provides a much finer characterization of task correlations in terms of layer-level features. Then, it is interesting to see that the learn-
  ing behavior for the three cases follows the same trend. This observation further corroborates that TRGP can improve the accuracy and mitigate forgetting on both “easy” and “difficult” tasks.
  Impact of selected tasks in trust region. We first evaluate the accuracy of Top-1 and Top-2 selected tasks as shown in Table 3. It shows that selecting the top-2 most correlated tasks could achieve better accuracy on both CIFAR-100 Split and 5-Dataset settings. Note that good performance can also be achieved even with the Top-1 case. Moreover, we illustrate the detailed task selection in the trust region for both layer-wise and task-wise on 5-Dataset setting in Fig. 7. For the task-wise, current task always selects the two adjacent previous tasks for all layers. Differently, the task selection varies for layer-wise, leading to more accurate selection of related tasks for each layer. For example, the layer wise trust region for Task 4 (Fashion MNIST) selects Task 1 (MNIST) or Task 3 (not-MNIST) as the most related tasks almost for all layers, over Task 0 (CIFAR-10) and Task 2 (SVHN), which clearly makes sense because Fashion MNIST shares more common features with MNIST and not-MNIST.
sections.heading: 6 CONCLUSION
sections.text: In this work, we propose trust region gradient projection for continual learning to facilitate forward knowledge transfer with forgetting, based on an efficient characterization of task correlation. Particularly, our approach is built on two key blocks, i.e., the layer-wise trust region which effectively select the old tasks strongly correlated to the new task in a single-shot manner, and scaled weight projection which cleverly reuses the frozen weights of old tasks in the trust region without modifying the model. Extensive experiments show that our approach significantly improves over the related state-of-the-art methods.
  REPRODUCIBILITY STATEMENT
  For the experimental results presented in the main text, we include the code in the supplemental material, and specify all the training details in Section 5.1 and Appendix A. For the datasets used in the main text, we also give a clear explanation in Section 5.1.
sections.heading: A EXPERIMENT SETUPS
sections.text: Training hyper-parameters. We evaluate our method on multiple datasets against state-of-the-art continual learning methods. 1) PMNIST. We use a 3-layer fully-connected network. with two hidden layer of 100 units. and train the network for 5 epochs with batch size of 10 for each task. 2) CIFAR-100 Split. CIFAR-100 (Krizhevsky et al., 2009) consists of images from 100 generic object classes. We use a version of 5-layer AlexNet and train each task for maximum of 200 epochs with the early termination strategy based on the validation loss value. The batch size is set to 64. 3) CIFAR100 Sup. We use a modified version of LeNet-5 with 20-50-800-500 neurons and train 50 epochs for each task sequentially. The batch size is set to 64. 4) 5-Datasets. We train each task for maximum of 200 epochs with the early termination strategy. The batch size is set to 64. 5) MiniImageNet Split. Following GPM (Saha et al., 2021), we use the reduced ResNet18 architecture, where the covolution with stride 2 in the first layer. We train each task for maximum of 100 epochs with the early termination strategy with 0.1 initial learning rate and 64 batchsize. In addition, for all the experiments, the threshold l is set to 0.5, and we select top-2 tasks that satisfy condition Eq. (3). We use the same threshold lth as GPM (Saha et al., 2021) for subspace construction. We initialize the scaling matrix with the identity matrix and train all models with plain stochastic gradient descent.
sections.heading: B MORE EXPERIMENTAL RESULTS
sections.text: B.1 ACCURACY EVOLUTION
  B.2 STANDARD DEVIATION
  We have summarized the results on the standard deviation for the averaged accuracy and backward transfer over 5 different runs on all datasets in Table 4.
  B.3 FORWARD TRANSFER
  To evaluate the forward transfer, we follow the metric used in (Veniat et al., 2020) and consider the accuracy of the model on i-th task after learning the i-th task sequentially, i.e., Ai,i as defined in Eq. (10). Tables 5 - 8 summarize the comparison of Ai,i for each task i between GPM and TRGP on
  PMNIST, CIFAR-100 Split and 5-Dataset, respectively. As the same baseline (e.g., the accuracy of the model learnt from scratch using the task’s own data) for each task will be used when evaluating the forward transfer for GPM and TRGP, we can infer that TRGP achieves the forward transfer gain of 0.17%, 2.01%, 2.00% and 2.36% over GPM on PMNIST, CIFAR-100 Split, 5-Datasets and MiniImageNet respectively.
  B.4 COMPUTATIONAL COMPLEXITY
  Memory: In terms of the memory, the major difference between TRGP and GPM is that TRGP requires additional memory to store the scaling matrices for each task. However, since the dimension of the scaling matrix is the same with the number of the extracted bases for the input subspace, which is usually small and controllable by the matrix approximation accuracy th in Eq. (7), the memory increase is marginal and controllable. Particularly, the memory usage of TRGP can be further reduced by only learning the scaling matrices for the convolutional layers.
  Training time: We compare the training time between TRGP and other baselines on relatively complex task sequences. As shown in Table 9, for CIFAR-100 Split, TRGP takes around 65% more
  time than GPM, is comparable with HAT and ER Res, and takes less time than OWM and EWC; for 5-Datasets, TRGP takes around 21% more time than GPM, but is much faster than other baselines including EWC, HAT, A-GEM and ER Res; for MiniImageNet, TRGP tasks around 34% more time than GPM, is comparable with EWC, but is much faster than A-GEM.
  B.5 ACCURACY VS LEARNING EPOCHS
  The learning dynamics for each task are shown in Figure 9 and 10. Clearly, our approach can perform significantly better than GPM on some tasks, especially for the tasks in the tail of the task sequence. This is because in GPM, with more tasks being learnt, the optimization space for new tasks becomes more restrictive, leading to limited performance for new tasks. Note that the y-axis is the validation accuracy with a split validate dataset that used during training, by following the setup in (Saha et al., 2021). The validation accuracy varies because the size of the validate dataset is relatively small (See (Saha et al., 2021) for the specific size). For the testing accuracy in all the tables, we evaluate the accuracy with the testing dataset after training.
title: TRGP: TRUST REGION GRADIENT PROJECTION