summaries: This paper proposes a new approach for continual learning that actively forgets the old knowledge when learning the new tasks. The approach uses Bayesian continual learning with synaptic expansion-convergence for active forgetting. The paper evaluates the new method on several continual learning benchmarks, including CIFAR10 regression, visual classification, and Atari reinforcement tasks. This paper investigates how to avoid negative transfer in Bayesian continual learning setting. The main contribution is the proposition of an active forgetting mechanisms inspired by biological neural networks for mitigating negative transfer in CL. The experiments on CL benchmarks have been conducted to validate the performances of the proposed method.  The paper proposed a continual learning method that attempts to promote better forward transfer among similar tasks. Towards this, the authors attempt to suppress the negative knowledge, the knowledge that impedes learning a new task, from the previous task while learning a new task. This is done by forgetting such knowledge from the previous task without increasing the forgetting catastrophically. The authors claim that such “active forgetting” is biologically inspired. Similar to the Elastic Weight Consolidation (EWC), the authors derive the learning objective, by applying the laplace approximation to the posterior of a new task. The only difference is that the posterior of the previous task is replaced by a weighted product distribution, where a weight of ‘\beta’ is introduced to capture active forgetting. This change allowed them to derive a new regularization penalty whereby a separate model is trained for a new task and then subsequently merged with that of the previous one. It is in this merging that the negative knowledge of the previous task is eliminated. Experimentally, the authors show strong performance on regression, classification and RL tasks. The authors further show that their method can be added to other regularization- and memory-based methods to improve their performance.   This paper proposes a method for continual learning that aims to mitigate the problem of catastrophic forgetting by actively forgetting knowledge from previous tasks. The authors use a Bayesian continual learning framework to design their method. AFEC relies on identifying a set of expanded network parameters which are specific to the new task. The method incorporates a term in the loss function that considers the merge between these parameters and the network parameters being updated for the current task, therefore encouraging active forgetting. Catastrophic forgetting is also controlled by including a term that merges between parameters for old tasks and network parameters for the current task (i.e. the EWC term). Experimental results are reported for datasets on regression, classification and reinforcement learning tasks. These results show some advantages of the method in particular in the forward transfer setting. 
abstractText: Continual learning aims to learn a sequence of tasks from dynamic data distributions. Without accessing to the old training samples, knowledge transfer from the old tasks to each new task is difficult to determine, which might be either positive or negative. If the old knowledge interferes with the learning of a new task, i.e., the forward knowledge transfer is negative, then precisely remembering the old tasks will further aggravate the interference, thus decreasing the performance of continual learning. By contrast, biological neural networks can actively forget the old knowledge that conflicts with the learning of a new experience, through regulating the learning-triggered synaptic expansion and synaptic convergence. Inspired by the biological active forgetting, we propose to actively forget the old knowledge that limits the learning of new tasks to benefit continual learning. Under the framework of Bayesian continual learning, we develop a novel approach named Active Forgetting with synaptic Expansion-Convergence (AFEC). Our method dynamically expands parameters to learn each new task and then selectively combines them, which is formally consistent with the underlying mechanism of biological active forgetting. We extensively evaluate AFEC on a variety of continual learning benchmarks, including CIFAR-10 regression tasks, visual classification tasks and Atari reinforcement tasks, where AFEC effectively improves the learning of new tasks and achieves the state-of-the-art performance in a plug-and-play way.
authors.name: Liyuan Wang
authors.name: Mingtian Zhang
authors.name: Zhongfan Jia
authors.name: Qian Li
authors.name: Kaisheng Ma
authors.name: Chenglong Bao
authors.name: Jun Zhu
authors.name: Yi Zhong
references.authors: Rahaf Aljundi Francesca Babiloni Mohamed Elhoseiny Marcus Rohrbach Tinne Tuytelaars
references.title: Memory aware synapses: Learning what (not) to forget
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Arslan Chaudhry Puneet K Dokania Thalaiyasingam Ajanthan Philip HS Torr
references.title: Riemannian walk for incremental learning: Understanding forgetting and intransigence
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Xinyang Chen Sinan Wang Bo Fu Mingsheng Long Jianmin Wang
references.title: Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Natalia Díaz-Rodríguez Vincenzo Lomonaco David Filliat Davide Maltoni
references.title: Don’t forget, there is more than forgetting: new metrics for continual learning
references.venue: arXiv preprint arXiv:1810.13166,
references.authors: Tao Dong Jing He Shiqing Wang Lianzhang Wang Yuqi Cheng Yi Zhong
references.title: Inability to activate rac1-dependent forgetting contributes to behavioral inflexibility in mutants of multiple autism-risk genes
references.venue: Proceedings of the National Academy of Sciences,
references.authors: Arthur Douillard Matthieu Cord Charles Ollion Thomas Robert Eduardo Valle
references.title: Podnet: Pooled outputs distillation for small-tasks incremental learning
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Geoffrey E Hinton
references.title: Training products of experts by minimizing contrastive divergence
references.venue: Neural Computation,
references.authors: Saihui Hou Xinyu Pan Chen Change Loy Zilei Wang Dahua Lin
references.title: Learning a unified classifier incrementally via rebalancing
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Sangwon Jung Hongjoon Ahn Sungmin Cha Taesup Moon
references.title: Continual learning with node-importance based adaptive group sparse regularization
references.authors: Steven M Kay
references.title: Fundamentals of statistical signal processing
references.venue: Prentice Hall PTR,
references.authors: James Kirkpatrick Razvan Pascanu Neil Rabinowitz Joel Veness Guillaume Desjardins Andrei A Rusu Kieran Milan John Quan Tiago Ramalho Agnieszka Grabska-Barwinska
references.title: Overcoming catastrophic forgetting in neural networks
references.venue: Proceedings of the National Academy of Sciences,
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning multiple layers of features from tiny images
references.venue: Technical report, Citeseer,
references.authors: Alex Krizhevsky Ilya Sutskever Geoffrey E Hinton
references.title: Imagenet classification with deep convolutional neural networks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Yann LeCun
references.title: The mnist database of handwritten digits. http://yann
references.venue: lecun. com/exdb/mnist/,
references.authors: Yaoyao Liu Bernt Schiele Qianru Sun
references.title: Meta-aggregating networks for class-incremental learning
references.venue: arXiv preprint arXiv:2010.05063,
references.authors: David Lopez-Paz
references.title: Gradient episodic memory for continual learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Michael McCloskey Neal J Cohen
references.title: Catastrophic interference in connectionist networks: The sequential learning problem
references.venue: In Psychology of Learning and Motivation,
references.authors: Cuong V Nguyen Yingzhen Li Thang D Bui Richard E Turner
references.title: Variational continual learning
references.venue: arXiv preprint arXiv:1710.10628,
references.authors: Sylvestre-Alvise Rebuffi Alexander Kolesnikov Georg Sperl Christoph H Lampert
references.title: icarl: Incremental classifier and representation learning
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Matthew Riemer Ignacio Cases Robert Ajemian Miao Liu Irina Rish Yuhai Tu Gerald Tesauro
references.title: Learning to learn without forgetting by maximizing transfer and minimizing interference
references.authors: Olga Russakovsky Jia Deng Hao Su Jonathan Krause Sanjeev Satheesh Sean Ma Zhiheng Huang Andrej Karpathy Aditya Khosla Michael Bernstein
references.title: Imagenet large scale visual recognition challenge
references.venue: International Journal of Computer Vision,
references.authors: John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov
references.title: Proximal policy optimization algorithms
references.venue: arXiv preprint arXiv:1707.06347,
references.authors: Jonathan Schwarz Wojciech Czarnecki Jelena Luketina Agnieszka Grabska-Barwinska Yee Whye Teh Razvan Pascanu Raia Hadsell
references.title: Progress & compress: A scalable framework for continual learning
references.venue: In Proceedings of International Conference on Machine Learning,
references.authors: Ramprasaath R Selvaraju Michael Cogswell Abhishek Das Ramakrishna Vedantam Devi Parikh Dhruv Batra
references.title: Grad-cam: Visual explanations from deep networks via gradientbased localization
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Hanul Shin Jung Kwon Lee Jaehong Kim Jiwon Kim
references.title: Continual learning with deep generative replay
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Yichun Shuai Binyan Lu Ying Hu Lianzhang Wang Kan Sun Yi Zhong
references.title: Forgetting is regulated through rac activity in drosophila
references.authors: Karen Simonyan Andrew Zisserman
references.title: Very deep convolutional networks for large-scale image recognition
references.venue: arXiv preprint arXiv:1409.1556,
references.authors: Shipeng Wang Xiaorong Li Jian Sun Zongben Xu
references.title: Training networks in null space of covariance for continual learning
references.venue: arXiv preprint arXiv:2103.07113,
references.authors: Shipeng Yan Jiangwei Xie Xuming He
references.title: Der: Dynamically expandable representation for class incremental learning
references.venue: arXiv preprint arXiv:2103.16788,
references.authors: Jaehong Yoon Saehoon Kim Eunho Yang Sung Ju Hwang
references.title: Scalable and order-robust continual learning with additive parameter decomposition
references.authors: Friedemann Zenke Ben Poole Surya Ganguli
references.title: Continual learning through synaptic intelligence
references.venue: In Proceedings of International Conference on Machine Learning,
sections.heading: 1 Introduction
sections.text: The ability to continually learn numerous tasks from dynamic data distributions is critical for deep neural networks, which needs to remember the old tasks by avoiding catastrophic forgetting [18] while effectively learn each new task by improving forward knowledge transfer [17]. Due to the dynamic data distributions, forward knowledge transfer might be either positive or negative, and is difficult to determine without accessing to the old training samples. If the forward knowledge transfer is negative, i.e., learning a new task from the old knowledge is worse than learning the new task on a randomly-initialized network [36, 17], then precisely remembering the old tasks will severely interfere with the learning of the new task, thus decreasing the performance of continual learning.
  ⇤Corresponding author: J. Zhu and Y. Zhong.
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  By contrast, biological neural networks can effectively learn a new experience on the basis of remembering the old experiences, even if they conflict with each other [18, 5]. This advantage, called memory flexibility, is achieved by active forgetting of the old knowledge that interferes with the learning of a new experience [29, 5]. The latest data suggested that the underlying mechanism of biological active forgetting is to regulate the learning-triggered synaptic expansion and synaptic convergence (Fig. 1, see Appendix A for neuroscience background and our biological data). Specifically, the biological synapses expand additional functional connections to learn a new experience together with the previously-learned functional connections (synaptic expansion). Then, all the functional connections are pruned to the amount before learning (synaptic convergence).
  Inspired by the biological active forgetting, we propose to actively forget the old knowledge that interferes with the learning of new tasks without significantly increasing catastrophic forgetting, so as to benefit continual learning. Specifically, we adopt Bayesian continual learning and actively forget the posterior distribution that absorbs all the information of the old tasks with a forgetting
  factor to better learn each new task. Then, we derive a novel method named Active Forgetting with synaptic Expansion-Convergence (AFEC), which is formally consistent with the underlying mechanism of biological active forgetting at synaptic structures. Beyond regular weight regularization approaches [12, 1, 35, 2], which selectively penalize changes of the important parameters for the old tasks, AFEC dynamically expands parameters only for each new task to avoid potential negative transfer from the main network, while the forgetting factor regulates a penalty to selectively merge the main network parameters with the expanded parameters, so as to learn a better overall representation of both the old tasks and the new task.
  We extensively evaluate AFEC on continual learning of CIFAR-10 regression tasks, a variety of visual classification tasks, and Atari reinforcement tasks [10], where AFEC achieves the state-of-theart (SOTA) performance. We empirically validate that the performance improvement results from effectively improving the learning of new tasks without increasing catastrophic forgetting. Further, AFEC can be a plug-and-play method that significantly boosts the performance of representative continual learning strategies, such as weight regularization [12, 1, 35, 2] and memory replay [21, 9, 6].
  Our contributions include: (1) We draw inspirations from the biological active forgetting and propose a novel approach to actively forget the old knowledge that interferes with the learning of new tasks for continual learning; (2) Extensive evaluation on a variety of continual learning benchmarks shows that our method effectively improves the learning of new tasks and achieves the SOTA performance in a plug-and-play way; and (3) To the best of our knowledge, we are the first to model the biological active forgetting and its underlying mechanism at synaptic structures, which suggests a potential theoretical explanation of how the underlying mechanism of biological active forgetting achieves its function of forgetting the past and continually learning conflicting experiences [29, 5].
sections.heading: 2 Related Work
sections.text: Continual learning needs to minimize catastrophic forgetting and maximize forward knowledge transfer. Existing work in continual learning mainly focuses on mitigating catastrophic forgetting. Representative approaches include: weight regularization [12, 1, 35, 2], which selectively penalizes changes of the previously-learned parameters; parameter isolation [24, 10], which allocates a dedicated parameter subspace for each task; and memory replay [21, 28, 9], which approximates and recovers the old data distributions through storing old training data, their embedding or learning a generative model. In particular, Adaptive Group Sparsity based Continual Learning (AGS-CL) [10] proposed to regularize the group sparsity with separation of the important nodes for the old tasks to prevent catastrophic forgetting, which takes advantages of weight regularization and parameter isolation, and achieved the SOTA performance on various continual learning benchmarks.
  Several studies suggested that forward knowledge transfer is critical for continual learning [17, 4], which might be either positive or negative due to the dynamic data distributions. Although it is highly nontrivial to mitigate potential negative transfer while overcoming catastrophic forgetting, the efforts that specifically consider this challenging issue are limited. [3] developed a method to mitigate negative transfer when fine-tuning tasks on a pretrained network. For the scenario where the old tasks can be learned again, [26] learned an additional active column to better exploit potential positive transfer. [22] tried to maximize transfer and minimize interference from a memory buffer containing a few old training data. Similarly, [6, 16, 33] attempted to more effectively balance stability and plasticity with the memory buffer in class incremental learning, while [32] stored and updated the old features. By contrast, since pretraining or old training data might not be available in continual learning, we mainly focus on a more restrict yet realistic setting that a neural network incrementally learns a sequence of tasks from scratch, without storing old training data. Further, we extend our method to the scenarios where pretraining or memory buffer can be used, as well as the scenarios other than classification tasks, such as regression tasks and reinforcement tasks.
sections.heading: 3 Method
sections.text: In this section, we first describe the framework of Bayesian continual learning [12, 20]. Under such framework, we propose an active forgetting strategy, which is formally consistent with the underling mechanism of biological active forgetting at synaptic structures.
sections.heading: 3.1 Basics of Bayesian Continual Learning
sections.text: Continual learning needs to remember the old tasks and learn each new task effectively. Let’s consider a simple case that a neural network with parameter ✓ continually learns two independent tasks, task A and task B, from their training datasets DtrainA and D train B [12]. The training dataset of each task is only available when learning the task.
  Bayesian Learning: After learning DtrainA , the posterior distribution
  p(✓|DtrainA ) = p(DtrainA |✓)p(✓)
  p(DtrainA )
  incorporates the knowledge of task A. Then, we can get the predictive distribution for the test data of task A:
  p(DtestA |DtrainA ) = Z p(DtestA |✓)p(✓|DtrainA )d✓.
  As the posterior p(✓|DtrainA ) is generally intractable (except very special cases), we must resort to approximation methods, such as the Laplace approximation [12] or other approaches of approximate inference [20]. Let’s take Laplace approximation as an example. If p(✓|DtrainA ) is smooth and majorly peaked around the mode ✓⇤A = argmax✓ log p(✓|DtrainA ), we can approximate it with a Gaussian distribution whose mean is ✓⇤A and covariance is the inverse Hessian of the negative log posterior (detailed in Appendix B.1).
  Bayesian Continual Learning: Next, we want to incorporate the new task into the posterior, which uses the posterior p(✓|DtrainA ) as the prior of the next task [12]:
  p(✓|DtrainA , DtrainB ) = p(DtrainB |✓) p(✓|DtrainA )
  p(DtrainB ) . (1)
  Then we can test the performance of continual learning by evaluating
  p(DtestA , D test B |DtrainA , DtrainB ) = Z p(DtestA , D test B |✓)p(✓|DtrainA , DtrainB )d✓. (2)
  Similarly, p(✓|DtrainA , DtrainB ) can be approximated by a Gaussian using Laplace approximation whose mean is the mode of the posterior:
  ✓⇤A,B = argmax ✓ log p(✓|DtrainA , DtrainB ) (3)
  = argmax ✓ log p(DtrainB |✓) + log p(✓|DtrainA ) log p(DtrainB )| {z } const. . (4)
  This MAP estimation is also known as the Elastic Weight Consolidation (EWC) [12]:
  LEWC(✓) = LB(✓) +
  2
  X
  i
  FA,i(✓i ✓⇤A,i)2, (5)
  where LB(✓) is the loss for task B and i is the label of each parameter. FA is the Fisher Information matrix (FIM) of ✓⇤A on D train A (the computation is detailed in Appendix B.1), which indicates the “importance” of parameter i for task A. The hyperparameter explicitly controls the penalty that selectively merges each ✓i to ✓⇤A,i to alleviate catastrophic forgetting.
sections.heading: 3.2 Active Forgetting with Synaptic Expansion-Convergence
sections.text: However, if precisely remembering task A interferes with the learning of task B, e.g., task A and task B are too different, it might be useful to actively forget the original data, similar to the biological strategy of active forgetting. Based on this inspiration, we introduce a forgetting factor and replace p(✓|DtrainA ) that absorbs all the information of DtrainA with a weighted product distribution [8, 19]:
  pm(✓|DtrainA , ) = p(✓|DtrainA )(1 )p(✓)
  Z , (6)
  where we use m to denote that we are ‘mixing’ p(✓|DtrainA ) and p(✓) to produce the new distribution pm. Z is the normalizer that depends on , which keeps pm(✓|DtrainA , ) following a Gaussian distribution if p(✓|DtrainA ) and p(✓) are both Gaussian (detailed in Appendix B.2). When ! 0, pm will be dominated by p(✓|DtrainA ) and remember all the information about task A. When ! 1, pm will actively forget all the information about task A. Modified from Eqn. (2), our target becomes:
  p(DtestA , D test B |DtrainA , DtrainB , ) = Z p(DtestA , D test B |✓)p(✓|DtrainA , DtrainB , )d✓. (7)
  We first need to determine , which decides how much information from task A is forgotten to maximize the probability of learning task B well. A good should be as follows:
  ⇤ = argmax p(DtrainB |DtrainA , ) = argmax Z p(DtrainB |✓)pm(✓|DtrainA , )d✓. (8)
  Since the integral is difficult to solve, we can make a grid search to determine , which should be between 0 and 1. Next, p(✓|DtrainA , DtrainB , ) can also be approximated by a Gaussian using Laplace approximation (the proof is detailed in Appendix B.3), and the MAP estimation is
  ✓⇤A,B = argmax ✓ log p(✓|DtrainA , DtrainB , )
  = argmax ✓
  (1 ) (log p(DtrainB |✓) + log p(✓|DtrainA )) + log p(✓|DtrainB ) + const.. (9)
  Then we obtain the loss function of Active Forgetting with synaptic Expansion-Convergence (AFEC):
  LAFEC(✓) =LB(✓) +
  2
  X
  i
  FA,i(✓i ✓⇤A,i)2 + e 2
  X
  i
  Fe,i(✓i ✓⇤e,i)2. (10)
  ✓⇤e are the optimal parameters for the new task and Fe is the FIM of ✓⇤e (the computation is detailed in Appendix B.1). As shown in Fig. 2, we first learn a set of expanded parameters ✓e with LB(✓e) to obtain ✓⇤e and Fe. Then we can optimize Eqn. (10), where two weight-merging regularizers selectively merge ✓i with ✓⇤A,i for the old tasks and ✓ ⇤ e,i for the new task. The forgetting factor is integrated into a hyperparameter e / /(1 ) to control the penalty that promotes active forgetting. Therefore, derived from active forgetting of the original posterior in Eqn. (6), we obtain an algorithm that dynamically expands parameters to learn a new task and then selectively converges the expanded parameters to the main network. Intriguingly, this algorithm is formally consistent with the underlying mechanism of biological active forgetting (the neuroscience evidence is detailed in Appendix A), which also expands additional functional connections for a new experience (synaptic expansion) and then prunes them to the amount before learning (synaptic convergence).
  As the proposed active forgetting is integrated into the third term, our method can be used in a plug-and-play way to improve continual learning (detailed in Appendix E, F). Here we use Laplace approximation to approximate the intractable posteriors, which can be other strategies of approximate inference [20] in further work. Note that ✓⇤e and Fe are not stored in continual learning, and the architecture of the main network is fixed. Thus, AFEC does not cause additional storage cost compared with regular weight regularization approaches such as [12, 35, 1, 2]. Further, it is straightforward to extend our method to continual learning of more than two tasks. We discuss it in Appendix B.4 with a pseudocode.
  Now we conceptually analyze how AFEC mitigates potential negative transfer in continual learning (see Fig. 2). When learning task B on the basis of task A, regular weight regularization approaches [12, 35, 1, 2] selectively penalize changes of the old
  parameters learned for task A, which will severely interfere with the learning of task B if they conflict with each other. In contrast, AFEC learns a set of expanded parameters only for task B to avoid potential negative transfer from task A. Then, the main network parameters selectively merge with both the old parameters and the expanded parameters, depending on their contributions to the overall representations of task A and task B.
sections.heading: 4 Experiment
sections.text: In this section, we evaluate AFEC on a variety of continual learning benchmarks, including: CIFAR10 regression tasks, which is a toy experiment to validate our idea about negative transfer in continual learning; visual classification tasks, where the forward knowledge transfer might be either positive or negative; and Atari reinforcement tasks, where the forward knowledge transfer is severely negative. All the experiments are averaged by 5 runs with different random seeds and task orders.
sections.heading: 4.1 CIFAR-10 Regression Tasks
sections.text: First, we propose CIFAR-10 regression tasks to explicitly show how negative transfer affects continual learning, and how AFEC effectively addresses this challenging issue. CIFAR-10 dataset [13] contains 50,000 training samples and 10,000 testing samples of 10-class colored images of size 32⇥ 32. The regression task is to evenly map the ten classes around the origin of the two-dimensional coordinates and train the neural network to predict the angle of the origin to each class (see Fig. 3). We change the relative position of the ten classes to construct different regres-
  sion tasks with mutual negative transfer, in which remembering the old knowledge will severely interfere with the learning of a new task.
  As shown in Fig. 4 for continual learning of two different regression tasks, regular weight regularization approaches, such as MAS [1] and EWC [12], can effectively remember the old tasks, but limits the learning of new tasks. In particular, larger strength of the weight regularization results in better performance of the
  first task but worse performance of the second task. In contrast, AFEC improves the learning of new tasks on the basis of remembering the old tasks, so as to achieve better averaged accuracy. Note that EWC is equal to the ablation of active forgetting in AFEC, i.e., = 0, so the performance improvement of AFEC on EWC validates the effectiveness of our proposal. We further demonstrate the efficacy of AFEC on a variety of architectures and a larger amount of tasks (see Table 1).
  In addition, we evaluate the ability of transfer learning after continual learning of two different regression tasks. We fix the feature extractor of the neural network and only fine-tune a linear classifier to predict a new task that is similar to the first task. Specifically, the similar task applies the same relative position as the first task, but rotates by 60 , 120 , 180 , 240 or 300 . Therefore, if the neural network effectively remembers and transfers the relative position learned in the first task, it will be able to learn the similar task well. As shown in Table 1, AFEC can more effectively learn the similar task, while EWC is even worse than sequentially fine-tuning without weight regularization.
sections.heading: 4.2 Visual Classification Tasks
sections.text: Dataset: We evaluate continual learning on a variety of benchmark datasets for visual classification, including CIFAR-100, CUB-200-2011 and ImageNet-100. CIFAR-100 [13] contains 100-class colored images of the size 32⇥ 32, where each class includes 500 training samples and 100 testing samples. CUB-200-2011 [31] is a large-scale dataset including 200 classes and 11,788 colored images of birds, split as 30 images per class for training while the rest for testing. ImageNet-100 [9] is a subset of iILSVRC-2012 [23], consisting of randomly selected 100 classes of images and 1300 samples per class. We follow the regular preprocessing pipeline of CUB-200-2011 and ImageNet-100 as [10], which randomly resizes and crops the images to the size of 224⇥ 224 before experiment. Benchmark: We consider five representative benchmarks of visual classification tasks to evaluate continual learning in different aspects. The first three are on CIFAR-100, with forward knowledge transfer from more negative to more positive (detailed in Fig. 5), while the second two are on largescale images. (1) CIFAR-100-SC [34]: CIFAR-100 can be split as 20 superclasses (SC) with 5 classes per superclass dependent on semantic similarity, where each superclass is a classification task. Since the superclasses are semantically different, forward knowledge transfer in such a task sequence is
  relatively more negative. (2) CIFAR-100 [21]: The 100 classes in CIFAR-100 are randomly split as 20 classification tasks with 5 classes per task. (3) CIFAR-10/100 [10]: The 10-class CIFAR-10 are randomly split as 2 classification tasks with 5 classes per task, followed by 20 tasks with 5 classes per task randomly split from CIFAR-100. This benchmark is adapted from [10] to keep the number of classes per task the same as benchmark (1, 2), where the large amounts of training data in the first two CIFAR-10 tasks bring a relatively more positive transfer. (4) CUB-200 [10]: The 200 classes in CUB-200-2011 are randomly split as 10 classification tasks with 20 classes per task. (5) ImageNet-100 [21]: The 100 classes in ImageNet-100 are randomly split as 10 classification tasks with 10 classes per task.
  Architecture: We follow [10] to use a CNN architecture with 6 convolution layers and 2 fully connected layers for benchmark (1, 2, 3), and AlexNet [14] for benchmark (4, 5). Since continual learning needs to quickly learn a usable model from incrementally collected data, we mainly consider learning the network from scratch. Following [10], we also try AlexNet with ImageNet pretraining for CUB-200.
  Baseline: First, we consider a restrict yet realistic setting of continual learning without access to the old training data, and perform multi-head evaluation [2]. Since AFEC is a weight regularization approach, we mainly compare with representative approaches that follow a similar idea, such as EWC [12], MAS [1], SI [35] and RWALK [2]. We also compare with AGS-CL [10], the SOTA method that takes advantage of weight regularization and parameter isolation, and P&C [26], which learns an additional active column on the basis of EWC to improve forward knowledge transfer. We reproduce the results of all the baselines from the officially released code of [10], where we do an extensive hyperparameter search and report the best performance for fair comparison (detailed in Appendix C). Then, we relax the restriction of using old training data and plug AFEC in representative memory replay approaches, where we perform single-head evaluation [2] (detailed in Appendix F).
  Averaged Accuracy: In Table 2, we summarize the averaged accuracy of all the tasks learned so far during continual learning of visual classification tasks. AFEC achieves the best performance on all the continual learning benchmarks and is much better than EWC [12], i.e., the ablation of active forgetting in AFEC. In particular, AGS-CL [10] is the SOTA method on relatively small-scale images and on CUB-200 with ImageNet pretraining (CUB-200 w/ PT). While, AFEC achieves a better performance than AGS-CL on small-scale images from scratch and CUB-200 w/ PT, and substantially outperforms AGS-CL on the two benchmarks of large-scale images from scratch. Further, since regular weight regularization approaches are generally in a re-weighted weight decay form, AFEC can be easily adapted to such approaches (the adaptation is detailed in Appendix E) and effectively boost their performance on the benchmarks above.
  Knowledge Transfer: Next, we evaluate knowledge transfer in the three continual learning benchmarks developed on CIFAR-100 in Fig. 5. We first present the accuracy of learning each new task
  in continual learning, where AFEC learns each new task much better than other baselines. Since continual learning of more tasks leads to less network resources for a new task, the overall trend of all the baselines is declining, indicating the necessity to improve forward knowledge transfer on the basis of overcoming catastrophic forgetting. Then we calculate forward transfer (FWT) [17], i.e., the averaged influence that learning the previous tasks has on a future task, and backward transfer (BWT) [17], i.e., the averaged influence that learning a new task has on the previous tasks (detailed in Appendix D). FWT is from more negative to more positive in CIFAR-100-SC, CIFAR-100 and CIFAR-10/100, while AFEC achieves the highest FWT among all the baselines. The BWT of AFEC is comparable as EWC, indicating that the proposed active forgetting does not cause additional catastrophic forgetting. Therefore, the performance improvement of AFEC in Table 2 is achieved by effectively improving the learning of new tasks in continual learning. In particular, AFEC achieves a much larger improvement on the learning of new tasks than P&C, which attempted to improve forward transfer of EWC through learning an additional active column. Due to the progressive
  parameter isolation, although AGS-CL achieves the best BWT, its ability of learning each new task drops more rapidly than other baselines. Thus, it underperforms AFEC in Table 2.
  Visual Explanation: To explicitly show how AFEC improves continual learning, in Fig. 6 we use Grad-CAM [27] to visualize predictions of the latest task after continual learning on CIFAR-100-SC, where FWT is more negative as discussed above. The predictions of EWC overfit the background information since it attempts to best remember the old tasks with severe negative transfer, which limits the learning of new tasks. In contrast, the visual explanation of AFEC is much more reasonable than EWC, indicating the efficacy of active forgetting to address potential negative transfer and benefit the learning of new tasks.
  Plugging-in Memory Replay: We further implement AFEC in representative memory replay approaches in Appendix F, where we perform single-head evaluation [2]. On CIFAR-100 and ImageNet100 datasets, we follow [9, 6] that first learn 50 classes and then continually learn the other 50 classes by 5 phases (10 classes per phase) or 10 phases (5 classes per phase), using a small memory buffer of 20 images per class. AFEC substantially boosts the performance of representative memory replay approaches such as iCaRL [21], LUCIR [9] and PODNet [6].
sections.heading: 4.3 Atari Reinforcement Tasks
sections.text: Next, we evaluate AFEC in continual learning of Atari reinforcement tasks (Atari games). We follow the implementation of [10] to sequentially learn eight randomly selected Atari games. Specifically, we applies a CNN architecture consisting of 3 convolution layers with 2 fully connected layers and identical PPO [25] for all the methods (detailed in Appendix G). The evaluation metric is the normalized accumulated reward: the evaluated rewards are normalized with the maximum reward of fine-tuning on each task, and accumulated. We present the results of three different orders of task sequence, averaged by five runs with different random initialization.
  For continual learning of Atari reinforcement tasks, forward knowledge transfer is severely negative, possibly because the optimal policies of each Atari games are highly different. We first measure the normalized rewards of learning each task with a randomly initialized network, which are 2.16, 1.44 and 1.68 on the three task sequences,
  respectively. That is to say, the initialization learned from the old tasks results in an averaged performance decline by 53.67%, 30.66% and 40.56%, compared with random initialization. Then, we evaluate the maximum reward of learning each new task in Table 3, and the normalized accumulated reward of continual learning in Fig. 7. AFEC effectively improves the learning of new tasks and thus boosts the performance of EWC and MAS, particularly when learning more incremental tasks. AFEC also achieves a much better performance than the reproduced results of AGS-CL on its officially released code [10] (see Appendix G for an extensive analysis).
sections.heading: 5 Conclusion
sections.text: In this work, we draw inspirations from the biological active forgetting and propose a novel approach to mitigate potential negative transfer in continual learning. Our method achieves the SOTA performance on a variety of continual learning benchmarks through effectively improving the learning of new tasks, and boosts representative continual learning strategies in a plug-and-play way. Intriguingly, derived from active forgetting of the past with Bayesian continual learning, we obtain the algorithm that is formally consistent with the synaptic expansion and synaptic convergence (detailed Appendix A), and is functionally consistent with the advantage of biological active forgetting in memory flexibility [5]. This connection provides a potential theoretical explanation of how the underlying mechanism of biological active forgetting achieves its function of forgetting the past and continually learning conflicting experiences. We will further explore it with artificial neural networks and biological neural networks in the future.
  Limitation and Social Impact
  The potential limitations of our work include three aspects: First, we propose a method to mitigate potential negative transfer in continual learning, so the efficacy of our method might be influenced by the level of negative transfer in a task sequence. Second, following [12], we assume that all the incremental tasks are independent, which might limit the application of our method to other scenarios such as continual learning of smoothly-changed data distributions. Third, our method needs to learn an additional set of parameters, resulting in more computational cost. Since our work is a fundamental research in machine learning, the negative social impacts are not obvious at this stage.
sections.heading: Acknowledgements
sections.text: This work was supported by NSF of China Projects (Nos. 62061136001, 61620106010, U19B2034, U181146, 62076145), Beijing NSF Project (No. JQ19016), Tsinghua-Peking Center for Life Sciences, Beijing Academy of Artificial Intelligence (BAAI), Tsinghua-Huawei Joint Research Program, a grant from Tsinghua Institute for Guo Qiang, and the NVIDIA NVAIL Program with GPU/DGX Acceleration.
title: AFEC: Active Forgetting of Negative Transfer in Continual Learning