summaries: This paper introduces a model for continual learning based on the decomposition of linear filters into low-rank components, called atoms. Specifically, the authors decompose convolutional filters shaped (c,c',k,k) into two components: i) alpha, shaped (c,c',m) and D, shaped (m,k,k). The former is learned on the first task and then frozen, whereas for D every task has its own and they are do not conflict during optimization. On top of that, the authors envision two different ensembling schemes that improve performances. i) First, in task-incremental settings, they retrieve atoms from the task of interest and from similar tasks as well (based on SVD decomposition of D matrices and Grassman distance) and ensemble them. ii) Furthermore, in class-incremental settings, they explicitly setup multiple atoms per task, building a task ensemble. During inference, all ensembles are queried and their predictive variance is used to "discover" the relevant task, for which a prediction is carried out. Experiments are carried out on 3 datasets in both task-incremental and class-incremental learning settings.  The paper proposes a continual learning algorithm that enforces the convolutional filter in each layer to a low-rank filter subspace defined by a small set of filter atoms. For each task, each convolutional layer is defined by a new filter subspace but subspace coefficients are shared among the tasks. The algorithm is validated on multiple benchmark datasets.  The paper, motivated by the task subspace modeling literature, enforced a low-rank filter structure to each CNN layer across time in continual learning. It not only ensures that the knowledge of the past tasks is not lost but also saves a lot of computing memory. Meanwhile, the paper proposes novel intra-task ensembles and inter-task ensembles for class-incremental settings and task-incremental settings, respectively.  This paper tackles the continual learning via enforcing a low-rank filter structure to each CNN layer.  They first perform atom-coefficient filter decomposition and then learn each task with a new filter subspace, so that the method only needs to save the new filters for each task. The contribution of this paper includes the low-rank filter scheme and the designed intra-task and inter-task model ensemble performing on the filters. The proposed method also achieves SOTA performance on several datasets with tiny size of model memory. 
abstractText: Continual learning is widely studied in recent years to resolve the catastrophic forgetting of deep neural networks. In this paper, we first enforce a low-rank filter subspace by decomposing convolutional filters within each network layer over a small set of filter atoms. Then, we perform continual learning with filter atom swapping. In other words, we learn for each task a new filter subspace for each convolutional layer, i.e., hundreds of parameters as filter atoms, but keep subspace coefficients shared across tasks. By maintaining a small footprint memory of filter atoms, we can easily archive models for past tasks to avoid forgetting. The effectiveness of this simple scheme for continual learning is illustrated both empirically and theoretically. The proposed atom swapping framework further enables flexible and efficient model ensemble with members selected within task or across tasks to improve the performance in different continual learning settings. The proposed method can be applied to a wide range of optimization schemes and convolutional network structures. Being validated on multiple benchmark datasets, the proposed method outperforms the state-of-the-art methods in both accuracy and scalability.
authors.name: Zichen Miao
authors.name: Ze Wang
authors.name: Wei Chen
authors.name: Qiang Qiu
references.authors: P-A Absil Robert Mahony Rodolphe Sepulchre
references.title: Riemannian geometry of grassmann manifolds with a view on algorithmic computation
references.venue: Acta Applicandae Mathematica,
references.authors: Alessandro Achille Michael Lam Rahul Tewari Avinash Ravichandran Subhransu Maji Charless C Fowlkes Stefano Soatto Pietro Perona
references.title: Task2vec: Task embedding for meta-learning
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Rahaf Aljundi Francesca Babiloni Mohamed Elhoseiny Marcus Rohrbach Tinne Tuytelaars
references.title: Memory aware synapses: Learning what (not) to forget
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Rahaf Aljundi Francesca Babiloni Mohamed Elhoseiny Marcus Rohrbach Tinne Tuytelaars
references.title: Memory aware synapses: Learning what (not) to forget
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Rahaf Aljundi Eugene Belilovsky Tinne Tuytelaars Laurent Charlin Massimo Caccia Min Lin Lucas Page-Caccia
references.title: Online continual learning with maximal interfered retrieval
references.venue: Advances in Neural Information Processing Systems,
references.authors: Rahaf Aljundi Min Lin Baptiste Goujaud Yoshua Bengio
references.title: Gradient based sample selection for online continual learning
references.venue: Advances in Neural Information Processing Systems,
references.authors: Eden Belouadah Adrian Popescu
references.title: Il2m: Class incremental learning with dual memory
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Francisco M Castro Manuel J Marín-Jiménez Nicolás Guil Cordelia Schmid Karteek Alahari
references.title: End-to-end incremental learning
references.venue: In Proceedings of the European conference on computer vision (ECCV),
references.authors: Arslan Chaudhry Puneet K Dokania Thalaiyasingam Ajanthan Philip HS Torr
references.title: Riemannian walk for incremental learning: Understanding forgetting and intransigence
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Arslan Chaudhry Puneet K Dokania Thalaiyasingam Ajanthan Philip HS Torr
references.title: Riemannian walk for incremental learning: Understanding forgetting and intransigence
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Arslan Chaudhry Marc’Aurelio Ranzato Marcus Rohrbach Mohamed Elhoseiny
references.title: Efficient lifelong learning with a-gem
references.venue: In International Conference on Learning Representations,
references.authors: Arslan Chaudhry Marcus Rohrbach Mohamed Elhoseiny Thalaiyasingam Ajanthan Puneet K Dokania Philip HS Torr M Ranzato
references.title: Continual learning with tiny episodic memories. 2019a
references.authors: Arslan Chaudhry Marcus Rohrbach Mohamed Elhoseiny Thalaiyasingam Ajanthan Puneet K Dokania Philip HS Torr Marc’Aurelio Ranzato
references.title: On tiny episodic memories in continual learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Matthias De Lange Rahaf Aljundi Marc Masana Sarah Parisot Xu Jia Ales Leonardis Gregory Slabaugh Tinne Tuytelaars
references.title: Continual learning: A comparative study on how to defy forgetting in classification
references.venue: tasks. arXiv preprint arXiv:1909.08383,
references.authors: Matthias Delange Rahaf Aljundi Marc Masana Sarah Parisot Xu Jia Ales Leonardis Greg Slabaugh Tinne Tuytelaars
references.title: A continual learning survey: Defying forgetting in classification tasks
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
references.authors: Prithviraj Dhar Rajat Vikram Singh Kuan-Chuan Peng Ziyan Wu Rama Chellappa
references.title: Learning without memorizing
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Simon Shaolei Du Wei Hu Sham M Kakade Jason D Lee Qi Lei
references.title: Few-shot learning via learning the representation, provably
references.venue: In International Conference on Learning Representations,
references.authors: Sayna Ebrahimi Mohamed Elhoseiny Trevor Darrell Marcus Rohrbach
references.title: Uncertainty-guided continual learning in bayesian neural networks
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,
references.authors: Sayna Ebrahimi Franziska Meier Roberto Calandra Trevor Darrell Marcus Rohrbach
references.title: Adversarial continual learning
references.venue: In Computer Vision–ECCV 2020: 16th European Conference,
references.authors: An Evgeniou Massimiliano Pontil
references.title: Multi-task feature learning
references.venue: Advances in neural information processing systems,
references.authors: Robert M Gray
references.title: Toeplitz and circulant matrices: A review
references.authors: Raia Hadsell Dushyant Rao Andrei A Rusu Razvan Pascanu
references.title: Embracing change: Continual learning in deep neural networks
references.venue: Trends in Cognitive Sciences,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Saihui Hou Xinyu Pan Chen Change Loy Zilei Wang Dahua Lin
references.title: Learning a unified classifier incrementally via rebalancing
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Ghassen Jerfel Erin Grant Thomas L Griffiths Katherine Heller
references.title: Reconciling meta-learning and continual learning with online mixtures of tasks
references.venue: Advances in Neural Information Processing Systems,
references.authors: James Kirkpatrick Razvan Pascanu Neil Rabinowitz Joel Veness Guillaume Desjardins Andrei A Rusu Kieran Milan John Quan Tiago Ramalho Agnieszka Grabska-Barwinska
references.title: Overcoming catastrophic forgetting in neural networks
references.venue: Proceedings of the national academy of sciences,
references.authors: James Kirkpatrick Razvan Pascanu Neil Rabinowitz Joel Veness Guillaume Desjardins Andrei A Rusu Kieran Milan John Quan Tiago Ramalho Agnieszka Grabska-Barwinska
references.title: Overcoming catastrophic forgetting in neural networks
references.venue: Proceedings of the national academy of sciences,
references.authors: Soheil Kolouri Nicholas Ketz Xinyun Zou Jeffrey Krichmar Praveen Pilly
references.title: Attention-based structuralplasticity
references.venue: arXiv preprint arXiv:1903.06070,
references.authors: Simon Kornblith Mohammad Norouzi Honglak Lee Geoffrey Hinton
references.title: Similarity of neural network representations revisited
references.venue: In International Conference on Machine Learning,
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning multiple layers of features from tiny images
references.authors: Abhishek Kumar Hal Daume III
references.title: Learning task grouping and overlap in multi-task learning
references.venue: International Conference on Machine Learning,
references.authors: Balaji Lakshminarayanan Alexander Pritzel Charles Blundell
references.title: Simple and scalable predictive uncertainty estimation using deep ensembles
references.venue: Advances in Neural Information Processing Systems,
references.authors: Yann LeCun Léon Bottou Yoshua Bengio Patrick Haffner
references.title: Gradient-based learning applied to document recognition
references.venue: Proceedings of the IEEE,
references.authors: Sang-Woo Lee Jin-Hwa Kim Jaehyun Jun Jung-Woo Ha Byoung-Tak Zhang
references.title: Overcoming catastrophic forgetting by incremental moment matching
references.venue: In Proceedings of the 31st International Conference on Neural Information Processing Systems,
references.authors: Xilai Li Yingbo Zhou Tianfu Wu Richard Socher Caiming Xiong
references.title: Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting
references.venue: In International Conference on Machine Learning,
references.authors: Zhizhong Li Derek Hoiem
references.title: Learning without forgetting
references.venue: IEEE transactions on pattern analysis and machine intelligence,
references.authors: David Lopez-Paz Marc’Aurelio Ranzato
references.title: Gradient episodic memory for continual learning
references.venue: Advances in neural information processing systems,
references.authors: Arun Mallya Svetlana Lazebnik
references.title: Packnet: Adding multiple tasks to a single network by iterative pruning
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Nicolas Y Masse Gregory D Grant David J Freedman
references.title: Alleviating catastrophic forgetting using contextdependent gating and synaptic stabilization
references.venue: Proceedings of the National Academy of Sciences,
references.authors: Andreas Maurer Massi Pontil Bernardino Romera-Paredes
references.title: Sparse coding for multitask and transfer learning
references.venue: In International conference on machine learning,
references.authors: John Milnor James D
references.title: Stasheff. Characteristic Classes.(AM-76)
references.venue: Volume 76,
references.authors: Ari Morcos Maithra Raghu Samy Bengio
references.title: Insights on representational similarity in neural networks with canonical correlation
references.venue: Advances in Neural Information Processing Systems,
references.authors: Cuong V Nguyen Yingzhen Li Thang D Bui Richard E Turner
references.title: Variational continual learning
references.venue: In International Conference on Learning Representations,
references.authors: Maria-Elena Nilsback Andrew Zisserman
references.title: Automated flower classification over a large number of classes
references.venue: Sixth Indian Conference on Computer Vision, Graphics & Image Processing,
references.authors: German I Parisi Ronald Kemker Jose L Part Christopher Kanan Stefan Wermter
references.title: Continual lifelong learning with neural networks: A review
references.venue: Neural Networks,
references.authors: Ameya Prabhu Philip HS Torr Puneet K Dokania
references.title: Gdumb: A simple approach that questions our progress in continual learning
references.venue: In European conference on computer vision,
references.authors: Qiang Qiu Xiuyuan Cheng Guillermo Sapiro
references.title: Dcfnet: Deep neural network with decomposed convolutional filters
references.venue: In International Conference on Machine Learning,
references.authors: Maithra Raghu Justin Gilmer Jason Yosinski Jascha Sohl-Dickstein
references.title: Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability
references.authors: Sylvestre-Alvise Rebuffi Alexander Kolesnikov Georg Sperl Christoph H Lampert
references.title: icarl: Incremental classifier and representation learning
references.venue: In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition,
references.authors: Matthew Riemer Ignacio Cases Robert Ajemian Miao Liu Irina Rish Yuhai Tu Gerald Tesauro
references.title: Learning to learn without forgetting by maximizing transfer and minimizing interference
references.venue: In International Conference on Learning Representations,
references.authors: Amanda Rios Laurent Itti
references.title: Closed-loop gan for continual learning
references.venue: International Joint Conference on Artificial Intelligence,
references.authors: Hippolyt Ritter Aleksandar Botev David Barber
references.title: Online structured laplace approximations for overcoming catastrophic forgetting
references.venue: In NeurIPS,
references.authors: Anthony Robins
references.title: Catastrophic forgetting, rehearsal and pseudorehearsal
references.venue: Connection Science,
references.authors: Bernardino Romera-Paredes Hane Aung Nadia Bianchi-Berthouze Massimiliano Pontil
references.title: Multilinear multitask learning
references.venue: In International Conference on Machine Learning,
references.authors: Syed Shakib Sarwar Aayush Ankit Kaushik Roy
references.title: Incremental learning in deep convolutional neural networks using partial network sharing
references.venue: IEEE Access,
references.authors: Jonathan Schwarz Wojciech Czarnecki Jelena Luketina Agnieszka Grabska-Barwinska Yee Whye Teh Razvan Pascanu Raia Hadsell
references.title: Progress & compress: A scalable framework for continual learning
references.venue: In International Conference on Machine Learning,
references.authors: Joan Serra Didac Suris Marius Miron Alexandros Karatzoglou
references.title: Overcoming catastrophic forgetting with hard attention to the task
references.venue: In International Conference on Machine Learning,
references.authors: Hanul Shin Jung Kwon Lee Jaehong Kim Jiwon Kim
references.title: Continual learning with deep generative replay
references.venue: In Proceedings of the 31st International Conference on Neural Information Processing Systems,
references.authors: Binh Tang David S Matteson
references.title: Graph-based continual learning
references.venue: In International Conference on Learning Representations,
references.authors: Michalis K Titsias Jonathan Schwarz Alexander G de G Matthews Razvan Pascanu Yee Whye Teh
references.title: Functional regularisation for continual learning with gaussian processes
references.venue: In ICLR,
references.authors: Oriol Vinyals Charles Blundell Timothy Lillicrap Daan Wierstra
references.title: Matching networks for one shot learning
references.venue: Advances in neural information processing systems,
references.authors: P. Welinder S. Branson T. Mita C. Wah F. Schroff S. Belongie P. Perona
references.title: Caltech-UCSD Birds 200
references.venue: Technical Report CNS-TR-2010-001, California Institute of Technology,
references.authors: Mitchell Wortsman Vivek Ramanujan Rosanne Liu Aniruddha Kembhavi Mohammad Rastegari Jason Yosinski Ali Farhadi
references.title: Supermasks in superposition
references.venue: Advances in Neural Information Processing Systems,
references.authors: Yue Wu Yinpeng Chen Lijuan Wang Yuancheng Ye Zicheng Liu Yandong Guo Yun Fu
references.title: Large scale incremental learning
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Jaehong Yoon Eunho Yang Jeongtae Lee Sung Ju Hwang
references.title: Lifelong learning with dynamically expandable networks
references.venue: In International Conference on Learning Representations,
references.authors: Jaehong Yoon Saehoon Kim Eunho Yang Sung Ju Hwang
references.title: Scalable and order-robust continual learning with additive parameter decomposition
references.venue: In International Conference on Learning Representations,
references.authors: Lu Yu Bartlomiej Twardowski Xialei Liu Luis Herranz Kai Wang Yongmei Cheng Shangling Jui Joost van de Weijer
references.title: Semantic drift compensation for class-incremental learning
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Amir R Zamir Alexander Sax William Shen Leonidas J Guibas Jitendra Malik Silvio Savarese
references.title: Taskonomy: Disentangling task transfer learning
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Friedemann Zenke Ben Poole Surya Ganguli
references.title: Continual learning through synaptic intelligence
references.venue: In International Conference on Machine Learning,
references.authors: Friedemann Zenke Ben Poole Surya Ganguli
references.title: Continual learning through synaptic intelligence
references.venue: In International Conference on Machine Learning,
references.authors: Junting Zhang Jie Zhang Shalini Ghosh Dawei Li Serafettin Tasci Larry Heck Heming Zhang C-C Jay Kuo
references.title: Class-incremental learning via deep model consolidation
references.venue: In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,
references.authors: Yu Zhang Qiang Yang
references.title: A survey on multi-task learning
references.venue: IEEE Transactions on Knowledge and Data Engineering,
sections.heading: 1 INTRODUCTION
sections.text: Humans keep acquiring new concepts without forgetting crucial ones in the past. To endow intelligent agents with the same ability of long-term knowledge accumulation, continual learning (CL) has been intensively studied in recent years. In continual learning, an agent learns from a sequence of tasks, with the goal of gaining knowledge of each new task while preserving the capacity for resolving the old ones, therefore to avoid catastrophic forgetting. The recent advances of CL mainly follow several directions. One popular category among them is to maintain an external memory of the original images (Robins, 1995; Rebuffi et al., 2017), synthesized images (Shin et al., 2017), or parameter gradients (Lopez-Paz & Ranzato, 2017) for archiving the past. These memory-based methods often suffer from heavy memory footprints, while still forgetting about the previous tasks to some extent.
  Motivated by the literature on subspace modeling of tasks (Evgeniou & Pontil, 2007; Maurer et al., 2013; Zhang & Yang, 2021; Romera-Paredes et al., 2013; Kumar & Daume III, 2012), in this paper, we propose to learn for each task a new filter subspace for each convolutional layer, i.e., hundreds of parameters as filter atoms, but keep subspace coefficients shared across tasks. In other words, in a CNN, we enforce a low-rank filter subspace by decomposing convolutional filters within each network layer over a small set of filter atoms. Then, we perform continual learning by simply swapping filter atoms for each task. The effectiveness of our approach is empirically validated and further explained theoretically with an excess risk bound analysis.
  With the proposed approach, we can faithfully remember the past by only maintaining a small footprint atom memory to archive task-specific filter atoms. Any previously learned CNN models can now be exactly recovered by multiplying the task-shared coefficients with the task-specific atoms, which can be retrieved efficiently from the atom memory. Thus, the introduced filter atom memory permits historical knowledge to be recalled with a guarantee against forgetting. While comparing with state-of-the-art memory-based CL methods, our approach requires only storing for each task
  ∗This work is supported by DARPA TAMI program under No. HR00112190038.
  some tiny size of filter atoms, which in total are typically much smaller than the size of exemplars in most memory-based methods (Rebuffi et al., 2017; Prabhu et al., 2020), and therefore potentially supports continual learning on a large scale.
  Our atom swapping continual learning framework can effectively support both inter-task and intra-task model ensemble to further enhance performance in different continual learning settings: First, intertask ensemble utilizes the relevant past knowledge to boost the present task performance. To ensure that ensemble with past models can affect the current task positively, we only select a relevant subset of past models. With a life-long learning scenario in mind, we choose to assess task relevancy simply based on the filter subspace distance, which can be on-the-fly computed here via the Grassmann distance (Absil et al., 2004) among task-specific filter subspaces. Second, intra-task ensemble can be adopted in class-incremental setting to help task prediction with the minimal-entropy criterion. Usually, ensemble members are instantiated as independent CNNs, and their learning and inference are conducted separately for dissimilarity. However, this will lead to significant increase in training and inference time and memory usage. We address this problem by creating within a task multiple virtual members in a single CNN model by simply maintaining several groups of filter atoms in each layer. In this way, different intra-task members are integrated into a single network, and its learning and inference can be conducted efficiently with group convolution.
  We validate our simple yet effective approach on several continual learning benchmarks such as MNIST, CIFAR100, and miniImageNet using both class-incremental and task-incremental settings, and observe competitive results against state-of-the-art methods on all benchmarks with far less memory usage.
  We summarize our contributions as follows,
  • We learn for each task a new filter subspace for each layer, and keep subspace coefficients shared across tasks.
  • We maintain a small footprint filter atom memory that can faithfully archive past knowledge with a guarantee against forgetting in a highly scalable way.
  • We adopt inter-task ensemble for the present task by recalling past models based on an on-the-fly calculated task relevancy in task-incremental settings.
  • We propose intra-task ensemble for class-incremental settings by creating multiple virtual members in a single CNN model through different groups of filter atoms per layer.
sections.heading: 2 MOTIVATION
sections.text: We are motivated by the literature on task subspace modeling (Evgeniou & Pontil, 2007; Maurer et al., 2013; Zhang & Yang, 2021; Romera-Paredes et al., 2013; Kumar & Daume III, 2012), where it is commonly assumed that task parameters lie in a low dimensional subspace, so that tasks can be modeled as a set of latent basis tasks and their linear combinations. The latent basis tasks and the respective linear combinations are often obtained via alternative optimization by fixing one and optimizing the other (Kumar & Daume III, 2012).
  In our continual learning setting, we model tasks using convolutional neural networks (CNNs). Following (Qiu et al., 2018), we decompose a convolutional filter Wi ∈ Rc×c
  ′×k×k for the i-th layer over m filter atoms Di ∈ Rm×k×k, linearly combined by coefficient αi ∈ Rc×c
  ′×m, where c and c′ are the number of input and output channels, k is the spatial size of each atom. This can be written as Wi = αi ×Di. Note that this decomposition distributes a filter Wi into two imbalanced parts: αi for channel mixing with mcc′ parameters, and light-weight Di for spatial filtering with only mk2 entries. In all, we use W = α×D to denote the filters decomposition in a model with l convolutional layers, where W = {Wi}li=1,α = {αi}li=1,D = {Di}li=1 indicate all filters, coefficients, and atoms respectively.
  It is easy to observe that, within each CNN layer, we can borrow from the task subspace modeling methodology by creating a set of latent basis tasks through filter atoms Di and their linear combinations as atom coefficients αi. The linear combination coefficients αi are learned on the first task jointly with the first group of atoms, and then atoms for subsequent tasks are alternatively optimized by fixing αi.
  As illustrated in the subsequent sections, this seemingly under-fitting method not only enables an efficient way to faithfully archive past models with a guarantee against forgetting, but also supports efficient inter-task and intra-task model ensemble to further improve the performance.
sections.heading: 3 METHODOLOGY
sections.text: We consider the problem of learning T tasks sequentially. Formally, we denote the data distribution that associates with the t-th task as Dt = (X t,Yt), t ∈ {1, 2, ..., T}, from which a dataset Dt = {xit,yit} Nt i=1 is sampled for training. The goal of continual learning is to minimize the statistical risk of all seen tasks given no access to data Dt from previous tasks t ≤ T (Delange et al., 2021): T∑
  t=1
  E(X t,Yt)[L(F t(X t; θ),Yt)], (1)
  where L denotes the risk function, F t(·; θ) is the model for task t with parameter θ. Continual learning with a guarantee against forgetting can be achieved by storing learned parameters entirely in an external memoryM = {θt}Tt=1 given F t(·; θ) = F(·; θt), thus any previous model can be completely recovered by retrieving the corresponding parameters from the memory. However, such a straightforward solution based on parameter memory suffers severely on its poor scalability due to the large size of modern deep neural networks and the potentially long task sequence. We address the scalability issue and achieve guaranteed non-forgetting by decomposing convolutional filters in a CNN into task-specific filter atoms and task-shared coefficients. Then only hundreds of parameters per task need to be stored in an atom memory to guarantee non-forgetting. This proposed approach allows efficient inter-task and intra-task ensemble to further boost performance.
sections.heading: 3.1 A SCALABLE APPROACH AGAINST FORGETTING
sections.text: In CNNs, catastrophic forgetting occurs when a model learned from a sequence of past tasks is updated in favor of the current task, resulting in significant performance degradation. A straightforward solution, as mentioned before, is to archive Wt in an external memoryM = {Wt}Tt=1, and the representation space of any previous model can be faithfully recalled by memory retrieval. However, as deep CNNs contain great amount of parameters in Wt, this simple solution scales poorly with the number of tasks T . On the other hand, storing part of parameters, or small subsets of data for parameter flashback cannot avoid forgetting completely (Sarwar et al., 2019; Yoon et al., 2018).
  This dilemma can be resolved with the proposed filter atom decomposition, as shown in Fig. 1. With the filter decomposition described in Sec. 2, by storing task-specific filter atoms Dt into memory, and enforcing a task-shared coefficients α, the model archives the entire knowledge for each time point.
  We refer to the memory for storing atoms as the atom memory,MD = {Dt}Tt=1,Dt ∈ Rm×k×k, with each Dt learned in the t-th task with empirical risk minimization,
  arg min Dt Nt∑ i=1 L(F(xti;α,Dt),yti). (2)
  The task-shared coefficients α are learned on the first task jointly with the first group of atoms,
  arg min D1,α N1∑ i=1 L(F(x1i ;α,D1),y1i ). (3)
  In this way, we can guarantee that the statistical risk for a previous task t at any time point remains the same as we can recall faithfully the past model by multiplying the stored atoms with the task-shared coefficients,
  E(X t,Yt)[L(F t(X t; θ),Yt)] = E(X t,Yt)[L(F(X t;α×Dt),Yt)]. (4)
  Atom memory scalability. The proposed atom memory stores a group of atoms per task, which is scalable with increasing number of tasks. Formally, consider a l-layer CNN model with associated filters W = {Wi}li=1. As mentioned in Sec. 2, each filter can be decomposed as Wi = αi ×Di. The group of atoms for task t can then be denoted as Dt = {Dti}li=1, which requires a size of lmk2 in storing parameters per task. This typically introduces only a few hundred of parameters for each task to be stored in the atom memory, which potentially supports continual learning on a large scale. Details for scalability comparison are shown in Sec. 5.2.1.
  Analysis of excess risk bound. With task-shared coefficients α, the model for each new task may seemingly expect some degree of underfitting. However, as demonstrated in Section 5, we still observe superior results over the state-of-the-art continual learning methods on all benchmarks we have evaluated. To understand this, we here theoretically analyze the excess risk bound for new tasks in the continual learning setting. For each tasks t ∈ {1, 2, ..., T}, model F t(·; θ) consists of representation function φt and prediction function wt (wt ∈ Rk). The representation function φt maps an input x to feature space Z ⊆ Rk. For the analysis purpose, we assume φt is just one convolution layer, which can be decomposed as φt = α×Dt. Therefore, using the training samples from task t, we can solve the following optimization problem from Eq. (2):
  min 1
  2N t ‖yt − (α×Dt(xt))+wt‖2 +
  λ 2 ‖α×Dt‖2F + λ 2 ‖wt‖2F , (5)
  where we use the mean square error as our loss, (·)+ is ReLU activation (z)+ = max{0, z}. The shared coefficients α are learned on the first task jointly with the first group of atoms. Similar to (Du et al., 2020), we also assume that there is a ground-truth optimal representation function φt,∗ and prediction function wt,∗ for task t.
  Assumption 1 (subgaussian input). There exists ρ > 0 such that, for all t ∈ {1, 2, ..., T}, the random vector x̄ ∼ p̄t is ρ2-subgaussian. The p̄t is the distribution of samples in task t. Assumption 2 (oracle network). Assume for task t ∈ {1, 2, ..., T} that yt = (α∗×Dt,∗(xt))+wt,∗+ zt is generated by an oracle network with parameters α∗, Dt,∗, and wt,∗. Noise term zt ∼ N (0, σ2I). Excess risk bound. We can bound the excess risk of our learned model on the task t, i.e., how much our learned model (α̂, D̂t, ŵt) performs worse than the optimal model (α∗,Dt,∗,wt,∗) on the task t as follows:
  ER(α̂, D̂t, ŵt) = LDt(α̂, D̂ t, ŵt)− LDt(α∗,Dt,∗,wt,∗) ≤ σR̄ · Õ( √ Tr(Σ) + √ ‖Σ‖2√
  N t ) + ρ4R̄2 · Õ(Tr(Σ) + ‖Σ‖2 N t )
  where R̄ = 12‖α ∗ ×Dt,∗‖2F + 12‖w t,∗‖2F , Σ = Ex∼p[xxᵀ], N t is the number of training samples. Õ is the big O notation, and LDt is the expected loss with the data distribution Dt. The detailed analysis is provided in the Appendix A.
sections.heading: 3.2 INTER-TASK MODEL ENSEMBLE
sections.text: The atom memory not only serves as an efficient way to archive past models with a guarantee against forgetting, but also enables efficient recall of past models for model ensemble to improve the performance in present task. As in Breiman (2001); Lakshminarayanan et al. (2017), ensemble performance increases with the independent level of ensemble members. Motivated by this, model ensemble over time for the current task at time r has a natural advantage that each ensemble member F t (t ∈ {1, 2, ..., r}) is learned from a different data distributionDt. However, most of other methods are not affordable to perform model ensemble across tasks since they often lack an effective and efficient way to recall past models. As illustrated in Fig. 2, with the atom memory, our method can faithfully and rapidly recall past models F t = F(·;α×Dt) by simply fetching atoms Dt from the atom memory; and then perform inter-task model ensemble by constructing a uniformly-weighted mixture model and combine the predictions as (Lakshminarayanan et al., 2017),
  Frc−ens(x) = 1 |Sr|+ 1 ∑
  s∈Sr∪{r}
  pF (y|x, θs) = 1
  Ec + 1 ∑ s∈Sr∪{r} F(x;α×Ds), (6)
  where Sr denotes the index set of previous tasks used for ensemble, and Ec = |Sr|. For classification problem, it corresponds to averaging the predictive probabilities.
  In continual learning, not all past models can bring positive effects on the current task. According to (Breiman, 2001; Lakshminarayanan et al., 2017), only the ones that have enough strengths on the current r-th task can help enhance the performance as weak learners (Breiman, 2001), which can be selected based on task relevancy. Our assumption is that the more a past task t resembles the current r, the better performance will F t achieves on the present task. The problem then transforms to evaluating the model similarity effectively and efficiently. Note that to ensure the proposed ensemble method to be scalable across a very long historical task sequence T , highly efficient task relevancy assessment is indispensable here.
  Assessing task relevancy by filter subspace distance. Task relevancy assessments proposed in (Achille et al., 2019; Zamir et al., 2018) work at the cost of heavy computation, which prevents their efficient applications in continual learning. While measuring the similarities among the learned models can be a straightforward proxy to the measurement of task relevancy, the widely studied methods, e.g., canonical correlation analysis (CCA) (Raghu et al., 2017; Morcos et al., 2018) and centered kernel alignment (CKA) (Kornblith et al., 2019), still introduce considerable computational cost while performing evaluations in the representation space. Directly performing relevancy measurements by calculating the distance of filters works at highly desirable efficiency, yet can perform poorly without the costly semantic alignments over channels (Raghu et al., 2017). In our approach, thanks to the task-shared coefficients acting as structural regularizations, we show that model similarity measurements can now be efficiently evaluated through direct filter similarity measurements. And as we model filters using task-specific filter subspaces with coefficients shared, model similarity can be further reduced to assessing filter subspace distance via the Grassmann distance (Absil et al., 2004).
  Formally, we characterize the filter subspace of the current model Vr and a past model Vt as: Vr = Span{Br1, ...,BrM}, Vt = Span{Bt1, ...,BtM}, Vr,Vt ⊂ RL, (7)
  where L = k2 is the dimension of kernel space, B = {Bj}Mj=1 are M (m > M ) linear independent vectors serve as the bases of the filter subspace. We obtain B by performing singular value decomposition (SVD) to atoms D = UΣV T ∈ RL×m and select the first M columns from U that correspond to the top-M singular values. By definition, Vr and Vt are points of the Grassmann Manifold (Milnor & Stasheff, 2016), i.e., Vr,Vt ∈ Grass(M,L) , {M dimensional subspaces of RL}. Then, the Grassmann distance between Vr and Vt is defined as,
  dM (Vr,Vt) = ( M∑ j θ2j ) 1/2, (8)
  where θj is the j-th principle angle, which can be calculated by,
  θj = arccos(σj), with (Br)TBt = UΣV T , σj = Σjj . (9)
  The proposed task relevancy measurement requires merely a SVD to matrices with dimensions lower than L. In practice, k = 3 so that L = 9, indicating that computation of dM is low. With the proposed efficient task relevancy measurement adopted in the last convolutional layer, we select the most Ec relevant models from previous r − 1 ones that support the current task. We provide the correlation analysis of task similarities measured with CCA (Raghu et al., 2017) and the ones assessed by the proposed subspace distance in Appendix. C.5.
  Although it is known that the ensemble result increases in the ensemble number Ec (Lakshminarayanan et al., 2017), it does not hold in our setting based on our empirical observation. In fact, determining the Ec is a trade-off between the amount and the relevancy of past knowledge. We thus empirically select the ensemble number Ec, which is illustrated in Sec. 5.2.1. The selected members are further fine-tuned with new classification heads to the current task.
sections.heading: 3.3 INTRA-TASK MODEL ENSEMBLE
sections.text: Our filter decomposition not only allows ensemble with past members, but also permits an efficient way to create ensemble members within a task. In the regular deep ensemble scenario (Lakshminarayanan et al., 2017), different members are instantiated as multiple CNNs, and need to be learned separately to ensure independence among members. Plus, obtaining ensemble results also requires inferences with multiple CNNs. This introduces significant cost in both time and memory, making it inappropriate in continual learning settings. The proposed atom decomposition allows a new way of parameterization of ensemble members within a task to improve performance while substantially reducing the training and testing time cost. Formally, given task t, the intra-task ensemble model F tw−ens is composed by {F̂ t,1, ..., F̂ t,Ew}, where Ew is the number of models. Rather than instantiating them as different CNN models, we reparameterize them with member-specific atoms {D̂t,1, ..., D̂t,Ew}, and member-shared coefficient αt, as shown in Fig. 3. In this way, we create multiple virtual members using a single CNN model by simply maintaining different groups of filter atoms in each convolutional layer. The forward pass of F tw−ens can then be conducted by group convolution,
  F tw−ens(xt) = 1
  Ew Ew∑ i=1 F̂ t,i(xt) = F tw−ens(x̂t, Ŵ tw−ens), (10)
  where x̂t ∈ Rn×(c×Ew)×h×w is the input repeated by Ew times, Ŵ tw−ens = [α × D̂t,1 | ... | α × D̂t,Ew ] ∈ Rc×(c′×Ew)×k×k (| denotes concatenation) is the filter for group convolution that is concatenated from per-member reconstructed filters. To enforce the independence of different members, we initialize member-specific atoms separately before training.
  With diverse predictions from different virtual members, our intra-task ensemble can directly boost the performance in task-incremental settings. Furthermore, our intra-task model ensemble makes the model to better distinguish data from out-of-task distribution, i.e., xp(p 6= t), and shows high entropy in its predictive distribution which is particularly useful in class-incremental (CI) setting. Note that in CI we perform task prediction first, select a specific group of atoms, and then perform classification within task. Task id is selected based on minimal-entropy criterion on predictive distributions. Thus, intra-task ensemble can enhance the accuracy of task prediction, and then the overall CI performance1.
  1Code is available at: https://github.com/ZichenMiao/CL_Atom_Swapping
sections.heading: 4 EXPERIMENTAL SETUP
sections.text: Datasets. We first validate our method on the Class-Incremental (CI) setting with CIFAR100 and ImageNet-Subset , which contains 100 classes selected from ImageNet (with random seed 1993). For each dataset, half of classes are used for model learning in the 0-th task. Then the remaining classes are equally split into N tasks, where N can be 5 or 10. Note that in the CI setting, task information is not provided during testing. Then, we report the performance of our method under the Task-Incremental (TI) setting. We validate our approach on 5-Split MNIST (LeCun et al., 1998), 20-Split CIFAR100 (Krizhevsky et al., 2009), and 20-Split miniImageNet (Vinyals et al., 2016). The 5-Split MNIST uniformly splits the original 10 classes of 0-9 MNIST digits into 5 sequential tasks. 20-Split CIFAR100 and 20-Split miniImageNet are both constructed by randomly splitting 100 classes into 20 tasks with 5 classes per task. Details of each dataset are provided in the Appendix C.
  Network architectures and implementation details. For the class-incremental setting, we utilize ResNet-32 for CIFAR100 and ResNet-18 for ImageNet-Subset as (Rebuffi et al., 2017). For the task-incremental setting, we adopt an AlexNet-like network. Note that we substitute all convolutional layers in both models with our decomposed version. Details of architecture are shown in the Appendix C. In terms of the proposed ensemble strategies, inter-task ensemble is only deployed in the task-incremental setting as task IDs are needed, while intra-task ensemble is utilized in both taskincremental and class-incremental settings. In inter-task ensemble, we set M = 3 for the dimension of filter subspaces. When intra-task ensemble and inter-task ensemble are adopted together, we use the member with the best results in every task for calculating task relevancy. We provide the ablation study for the inter-class and intra-class ensemble numbers Ec and Ew in Sec. 5.2.1. Training details are showed in the Appendix C.3. For forward knowledge transfer, the most recent atoms are used to initialize the atoms of the current task.
  Evaluation metrics. In the class-incremental setting, we evaluate the model’s average classincremental accuracy. In the task-incremental setting, we measure the performances with ACC as the average test accuracy across all tasks. To measure the forgetting, we adopt the backward transfer, BWT, which shows how the previous tasks performance has degraded due to learning new tasks. Details of these two measurements are provided in Appendix C.4.
sections.heading: 5 RESULTS AND DISCUSSION
sections.text: In this section, we start from the challenging class-incremental setting. Then we move to taskincremental setting with self-comparison experiments to validate the effectiveness of some key ingredients of our methods, and show the results of the proposed method on several real-world
  datasets. In both settings, our method achieves improvements over state-of-the-art methods with significant less memory usage.
sections.heading: 5.1 CLASS-INCREMENTAL EXPERIMENTS
sections.text: As a more challenging setting, class-incremental (CI) learning does not provide task id during testing. As mentioned in Sec. 3.3, we handle this setting by breaking it down to a two-level task, task prediction based on minimal-entropy criterion, and then within-task classification. In CI experiments, we select the number of atoms m = 12 and the number of members for intra-task ensemble Ew = 2. We benchmark our method by comparing with many existing methods that store 2000 exemplars in the external memory. As shown in Tab. 1, our method with intra-task ensemble achieves the best results with an order of magnitude less memory usage on both CIFAR100 and ImageNet-Subset, which validates the effectiveness and scalability of our framework in the challenging CI setting.
sections.heading: 5.2 RESULTS ON TASK-INCREMENTAL SETTINGS
sections.text: 5.2.1 SELF COMPARISONS ON 20-Split CIFAR100 In this section, we empirically analyze inter-task ensemble, intra-task ensemble, and selection of atoms of our method in the task-incremental setting. We analyze the performance of our base model with different number of atoms m, As shown in the left of Fig. 4, m = 12 is the best choice in terms of both performance and efficiency. We then test the task relevancy assessment based on subspace distance, and its instructive effect to inter-task ensemble. As shown in Fig. 5, the past models with small Grassmann distance to the current model lead to performance improvement by model ensemble. And past models with large Grassmann distance to the current one, in fact result in degraded performances. Note that we also present an ablation study on different metrics in Appendix C.8, which shows the superior performance of Grassmann distance over L2 and cosine distance. In general, we test the ACC of model ensemble with the top-1 to top-3 relevant model as well as the most irrelevant model starting from the 5-th task. As shown in Fig. 5, model ensemble with top-3 relevant models achieves the best results, and thus we set Ec = 3 in subsequent experiments. We then explore intra-task ensemble with different members Ew = 2, 3, 4. As illustrated in the middle of Fig. 4, ensemble within task enhances the performance consistently, and we chooseEw = 2 for the best performance efficiency trade-off. We further illustrate the scalability of the proposed method on the right of Fig. 4. Compared to PNN (Rusu et al., 2016) and APD (Yoon et al., 2019), the size of our method scales much slower with the number of tasks, even with ensemble adopted.
sections.heading: 5.2.2 COMPARISONS WITH BENCHMARKS
sections.text: We further report our results on 20-Split miniImageNet and 20-Split CIFAR100 in Tab. 2. Comparing with regularization-based, memory-based, and expansion-based models, the proposed method achieves the best results even with the base model. Adopting inter-task ensemble with the top-3 relevant past models, along with intra-task ensemble with 2 members, our method achieves further improved results. Especially in 20-Split miniImageNet, the previous state-of-the-art method, ACL (Ebrahimi et al., 2020) achieves ACC of 62.07 with a memory in size of 8.5 MB; whereas the proposed method improves the results significantly to 67.84 with a merely 0.86 MB memory. When comparing with other memory-based methods besides ACL, the proposed method demonstrates superior scalability reflected by the much smaller memory size. We provide additional results on the standard 5-Split MNIST dataset with our base model. As shown in Tab. D, our base method outperforms both regularization-based methods and memory-based methods in terms of ACC with much smaller memory. The improvements over the state-of-the-art methods and the outstanding scalability validate the effectiveness of our method on solving real-world continual learning problems.
sections.heading: 6 RELATED WORK
sections.text: Catastrophic forgetting is the central challenge of continual learning. Recent advances on continual learning is mainly driven by three main directions, which we will briefly review in terms of regularization-based, memory-based and expansion-based methods. For a more extensive overview we refer to the reviews by (De Lange et al., 2019; Parisi et al., 2019).
  Regularization-based methods. (Kirkpatrick et al., 2017b; Aljundi et al., 2018a; Lee et al., 2017; Zenke et al., 2017b; Kolouri et al., 2019) determine the importance of each model’s parameter per task, which prevents the important parameters from being updated for new tasks. For example, (Kirkpatrick et al., 2017b) specify the performance of each weight with the Fisher information matrix. (Aljundi et al., 2018a), on the other hand, proposes determining parameter importance by gradient magnitude. Theses methods can be naturally explored from the lens of Bayesian optimization (Nguyen et al., 2018; Titsias et al., 2020; Schwarz et al., 2018; Ebrahimi et al., 2019; Ritter et al., 2018). Specifically, (Nguyen et al., 2018) introduces a regularization technique, inspired by variational inference, to prevent their model from forgetting. All these methods address catastrophic forgetting by adding regularization terms. As pointed out in (De Lange et al., 2019), the penalty term proposed in such methods are unable to prevent drifts in the loss landscape of previous tasks. While alleviating forgetting, the penalty also unavoidably prevents the plasticity to absorb new in- formation from future tasks learned over a long timescale (Hadsell et al., 2020).
  Memory-based methods. (De Lange et al., 2019) assume it is feasible to access data from previous tasks by having a fixed-size memory or a generative model able to produces samples from previous tasks (Lopez-Paz & Ranzato, 2017; Riemer et al., 2018; Rios & Itti, 2018; Shin et al., 2017). (Rebuffi et al., 2017) introduces models augmented with fixed-size memory, which accumulates samples in the proximity of class centers. (Chaudhry et al., 2019b) proposes another memory-based model by exploiting a reservoir sampling strategy in the raw input data selection phase. Rather than storing the original samples, (Chaudhry et al., 2018a) accumulates the parameter gradients during task learning. (Shin et al., 2017) incorporate a generative model into a continual learning model to alleviate catastrophic forgetting by producing samples from previous task and retraining the model using data from previous tasks and the the current one. These methods assume an extra neural network, such as a generative model or a memory. Different from replay-based methods, which benefit from a memory to retrain their model over previous tasks, our method requires storage of tiny atoms for each previous task only, which is more scalable and do not suffer from potential forgetting caused by the inconsistent memory reply in generative-based methods.
  Expansion-based methods. (Rusu et al., 2016; Yoon et al., 2018; Jerfel et al., 2019; Li et al., 2019) allocate a subset of the model parameters for each task. Model expansion can be achieved by a gating mechanism (Wortsman et al., 2020; Masse et al., 2018), or by incrementally adding new parameters to the models (Rusu et al., 2016). Incrementally learning and pruning provides another direction (Mallya & Lazebnik, 2018). Given an over-parametrized model with the ability to learn potentially many tasks, (Mallya & Lazebnik, 2018) achieves model expansion by pruning the parameters not contributing to the performance of the current task, while keeping them avail- able for future tasks. Comparing to the aforementioned methods, the proposed method provides a novel view of subspace of deep convolutional neural networks, which allows model expansion to be achieved with significantly decreased parameters, and is therefore scalable to even thousands of tasks.
sections.heading: 7 CONCLUSION
sections.text: In this paper, motivated by the task subspace modeling literature, we enforced a low-rank filter structure to each CNN layer across time in continual learning. By performing atom-coefficient filter decomposition, we learned for each task a new filter subspace for each layer, while keeping subspace coefficients shared across tasks. This simple method allows highly efficient model storage and retrieval using a small footprint atom memory. The proposed method provided a guarantee against forgetting, and we demonstrated further performance improvements through model ensemble. The performance was evaluated on various continual learning tasks, and the effectiveness and scalability were demonstrated by the state-of-the-art accuracy and the tiny size of model memory.
sections.heading: APPENDIX
sections.heading: A EXPECTED EXCESS RISK
sections.text: Problem setup: Suppose we have T tasks sequentially. The data distribution Dt = (X t,Yt) associates t-th task (t ∈ {1, 2, ..., T}). For each task t ∈ [T ], we assume φt : X −→ Z is a representation function, which maps input images to feature space Z ⊆ Rk. These representation functions are restricted in function class Φ, such as neural networks. There is also a predictor wt : Z −→ Y for task t ∈ [T ] mapping the feature representation to labels. Our object is to minimize empirical risk:
  E(X t,Yt)[L(F t(X t; θ),Yt)].
  For analysis purpose, we consider representation function φt as single convolutional layer, which can be decomposed as φt = α×Dt. With the predictor wt, the loss can be written as:
  L(F t(X t; θ),Yt) = 1 2N t Nt∑ i=1 (yi − 〈wt, (α×Dt(xi))+〉)2,
  where Dt(x) is the convolution operation on the input images, (·)+ is ReLU activation. As the input x ∈ Rc×h×w and atoms D ∈ Rm×k×k, where h× w are the size of input images, c is the number of input channel, and m is the number of kernels which have size of k × k. Dt(x) contains c×m convolution operations that maps input x into the latent zl ∈ Rcm×h ′×w′ . The latent zl then maps into feature space Z by α× zl.
  We can treat each convolution operation as matrix product by converting the convolution kernel to a doubly blocked Toeplitz matrix D′t ∈ Rhw×h
  ′w′ (Gray, 2006), and converting input image x into x′ ∈ Rc×hw. Therefore, the convolution operation Dt(x) is transformed as reshaped image x′ multiple by m reformed kernels D′t. To simplify the problem, we assume consider each channel of the input images and each base of atoms, such that c = 1 and m = 1. Consider of all N t samples Xt ∈ RN t×hw with labels Y t ∈ RN t
  for task t, Thus, the loss can be written as:
  L(F t(X t; θ),Yt) = 1 2N t Nt∑ i=1 (yi − 〈wt, (α×Dt(xi))+〉)2
  = 1 2N t ‖Y t − (αᵀXtD′t)+wᵀt‖2
  The first task trains the α, D′1, and w1 at the same time, but the following tasks only optimize over Dt and wt and keep α fixed. With fixed α, we can represent the samples as X̃t = αᵀXt. By solving the object, we get (D̂t, ŵt). We also assume that there is an optimal atom Dt,∗ of representation function φt,∗ and prediction function wt,∗ for task t. We now analyze the bound for the excess risk of our learned model on the task t, i.e., how much our learned model (D̂t, ŵt) performs worse than the optimal model (Dt,∗,wt,∗) on the task t:
  ER(φ̂t, ŵt) = LDt(D̂t, ŵ t)− LDt(Dt,∗,wt,∗)
  = 1 2 ((X̃tD̂′ᵀt)+ŵ
  ᵀt − (X̃tD′ᵀt,∗)+wᵀt,∗)2 (1)
  A standard lifting of neural networks can be formulated as infinite dimension linear regression (Du et al., 2020). Define the infinite feature vector with the coordinates φ(x)b = (bᵀx)+ for every b ∈ Sd0−1. Let βt be a signed measure on Sd0−1. The inner product notation denotes integration: βᵀφ(x) = ∫ Sd0−1 φ(x)bdβb. By convert input feature x to the lifted feature vector φ(x), according to (Du et al., 2020), Equation. 1 can be bounded by: R(φ̂t, ŵt) = 1
  2 (X̃tD̂′ᵀtŵᵀt − X̃tD′ᵀt,∗wᵀt,∗)2.
  Let E[x̃x̃ᵀ] = Σ̃, we have 1
  2 (X̃tD̂′ᵀtŵᵀt − X̃tD′ᵀt,∗wᵀt,∗)2 = 1 2 ‖Σ̃1/2D′ᵀt,∗wᵀt,∗ − Σ̃1/2D̂′ᵀtŵᵀt‖2.
  With Assumption 1 and 2, based on the inequality in (Du et al., 2020), for a fixed δ > 0, we further have 1
  2 ‖Σ̃1/2D′ᵀt,∗wᵀt,∗ − Σ̃1/2D̂′ᵀtŵᵀt‖2 ≤ 2ee,1R̄+ 2ic,1R̄2 + 2ee,2r + 2ic,2r2,
  where 2ee,1 = 1√N1 σ(log 1 δ )3/2log(n) √ ‖Σ̃‖+ Tr(Σ̃), 2ee,2 = 1√Nt σ(log 1 δ )3/2log(n) √ ‖Σ̃‖ are estimation error and ic,1 = 2Cρ 2
  √ N1
  ( √ Tr(Σ̃) + √ log 2
  δ ‖Σ̃‖), ic,2 = Cρ 2 √ Nt
  √ ‖Σ̃‖R̄log 1
  δ are intrinsic dimension concen-
  tration error, R̄ = 1 2 ‖α∗ ×Dt,∗‖2F + 12‖w t,∗‖2F . Therefore, suppose we have the same number of samples
  in each task, with probability at least 1− δ over the samples, the expected excess risk of the learned atom and predictor on the task satisfies:
  E[ER(φ̂t, ŵt)] ≤ σR̄ · Õ( √ Tr(Σ) + √ ‖Σ‖2√
  N t ) + ρ4R̄2 · Õ(Tr(Σ) + ‖Σ‖2 N t )
sections.heading: B ALGORITHM
sections.text: We provide the algorithm of the proposed method in Alg. 1.
  Algorithm 1 Continual Learning with Filter Atom Swapping
  Initialize MD ←− [ ], α = {αi}li=1 ←− α0, D1 = {D1i }li=1 ←− D1,0. for task t = 1, 2, ..., T do
  if t == 1 then Optimize α,D1 according to (3). Update Atom Memory MD ←− [MD,D1]. else if t < 5 then
  Initialize Dt = {Dti}li=1 ←− Dt,0, then optimize Dt according to (2). Update Atom Memory MD ←− [MD,Dt]. else Initialize Dt = {Dti}li=1 ←− Dt,0, then optimize Dt according to (2). Initialize distance list dt = [0, ..., 0],where length(dt) = t− 1. for j=1:t-1 do
  Calculate dM (Dtl ,D j l ) in the last convolutional layer according to (8), and update dt[j] = dM (D t l ,D j l )
  end for Sort dt in ascending order, and select least Ec indices to construct St. Finetune additional n heads on current task. Update Atom Memory MD ←− [MD,Dt].
  end if end if
  end for return MD, α.
sections.heading: C EXPERIMENTAL DETAILS
sections.heading: C.1 DATASET STATISTICS
sections.text: We provide the dataset statistics of 10-Split CIFAR100 used in class-incremental setting in Tab. A, and statistics of 20-Split CIFAR100and 20-Split miniImageNetin task-incremental setting in Tab. B and Tab. C.
  Table A: Statistics of 10-Split CIFAR100.
  10-Split CIFAR100 # tasks 10
  Img. Size 32× 32× 3 # tasks/task 10
  # Training samples/task 4500 # Validation samples/task 500
  # Test samples/task 100
sections.heading: C.2 NETWORK ARCHITECTURE
sections.heading: C.2.1 ARCHITECTURE FOR CLASS-INCREMENTAL LEARNING
sections.text: We adopt the ResNet-32 as in (Rebuffi et al., 2017), which includes 31 convolutional layers and 1 fully connected layer. We substitute every convolutional layer with the one has the proposed decomposed filters (m = 12) that has the same number of input/ output channels.
  Table B: Statistics of 20-Split CIFAR100and 20-Split miniImageNet.
  20-Split CIFAR100 20-Split miniImageNet # tasks 20 20
  Img. Size 32× 32× 3 84× 84× 3 # tasks/task 5 5
  # Training samples/task 2,125 2,125 # Validation samples/task 375 375
  # Test samples/task 500 500
  Table C: Statistics of 5-Split MNIST.
  Task (0,1) (2,3) (4,5) (6,7) (8,9) # Training samples/task 10,766 10,276 9,574 10,356 10,030
  # Validation samples/task 1,899 1,813 1,689 1,827 1,770 # Test samples/task 2,115 2,042 1,874 1,986 1,983
sections.heading: C.2.2 ARCHITECTURE FOR TASK-INCREMENTAL LEARNING
sections.text: Architecture for 5-Split MNIST. Conv(k=3, m=12, 16)-ReLU-Dropout(0.2)-MaxPool(2)-Conv(k=3, m=12,32)-ReLU-Dropout(0.2)-MaxPool(2)-Conv(k=3, m=12, 64)-ReLU-Dropout(0.2)-GAP-FC(32)-ReLUDropout(0.5)-FC(2),
  where GAP stands for global average poling, Conv(k, m, c) means the convolution layer with kernel size k, m atoms, c output channels, Dropout(p) indicates the dropout layer with probability p, and FC(c) is the fully-connect layer with c output channels.
  Architecture for 20-Split CIFAR100and 20-Split miniImageNet. Conv(k=3, m=12, 64)-ReLUMaxPool(2)-Conv(k=3, m=12, 192)-ReLU-MaxPool(2)-Conv(k=3, m=12, 384)-ReLU-Conv(k=3, m=12, 256)- ReLU-Conv(k=3, m=12, 256)-ReLU-MaxPool(2)-GAP-Dropout(0.5)-FC(5)
sections.heading: C.3 TRAINING DETAILS
sections.heading: C.3.1 CLASS-INCREMENTAL LEARNING
sections.text: For CIFAR100, we choose SGD with batch-size of 128, learning rate of 0.01, momentum of 0.9 and weight decay 1e-3. The model is trained for 250 epochs, with learning rate drop by 0.1 at the 100-th and 200-th epoch. For ImageNet-Subset, we choose SGD with batch-size of 128, learning rate of 0.05, momentum of 0.9, and weight decay 1e-4. The model is trained for 150 epochs, with learning rate drop by 0.1 at the 90-th and 120-th epoch. For CUBS and Flowers, We choose SGD with momentum 0.9, batch-size of 64, and weight decay of 1e-4. On CUBS, the model is finetuned for 200 epochs with learning rate of 1e-2 (drop by 0.1 at the 100th and 160th epoch) on the first task, and trained for 120 epochs with learning rate of 5e-3 (drop by 0.1 at the 70th epoch) on task 2-6. On Flowers, the model is finetuned for 120 epochs with learning rate of 1e-2 (decay by 0.1 at the 50th and 90th epoch) on the first task, and trained for 70 epochs with learning rate of 1e-3 (decay by 0.1 at the 40th epoch) on task 2-6.
sections.heading: C.3.2 TASK-INCREMENTAL LEARNING
sections.text: We choose SGD with batch-size of 64 for all experiments under the task-incremental setting. For 5-Split MNIST, we train the model for 20 epochs with learning rate of 0.001. For 20-Split CIFAR100, we train the model for 100 epochs with learning rate of 0.001, which drops by 0.1 at the 60-th epoch. For 20-Split miniImageNet, we train the model for 200 epochs with learning rate of 0.001, which drops at the 100-th and 150-th epoch.
sections.heading: C.4 METRIC
sections.text: ACC and BWT used in the task-incremental settings can be expressed as,
  ACC = 1
  T T∑ i=1 AccT,i, BWT = 1 T − 1 T−1∑ i=1 AccT,i − Acci,i, (2)
  where T is total number of tasks, Acci,j is performance on j-th task after learning i-th task (Lopez-Paz & Ranzato, 2017).
  Figure A: Correlation analysis between task similarities calculated with CCA (Raghu et al., 2017) and task similarities assessed with filter subspace distance.
sections.heading: C.5 CORRELATION ANALYSIS BETWEEN CCA AND FILTER SUBSPACE DISTANCE.
sections.text: We conduct the analysis between the task similarities measured with CCA (Raghu et al., 2017) on deep feature spaces, and the proposed task similarities assessed with Grassmann distance on filter subspaces. The analysis is conducted on 20-Split CIFAR100with the proposed model. We randomly select 50 task pairs from ( 20 2 ) task pairs. Given a task pair, we use 500 images from each task to calculate the CCA similarity, and use the inverse of their Grassmann distance between their last layers’ filter subspaces as the filter-subspace similarity. As shown in Fig. A, filter-subspace similarities quite correlates to CCA similarities which we deem as the golden standard, while they are significantly more efficient to calculate with our atom continual learning framework.
sections.heading: C.6 ADDITIONAL TASK-INCREMENTAL EXPERIMENTAL RESULTS
sections.text: We provide additional results on the standard 5-Split MNIST dataset. Note that due to the simplicity of 5-Split MNIST, we only apply on it our base model, which is composed by cross-task coefficient sharing and atom swapping only. As shown in Tab. D, our base method outperforms both regularization-based methods and memory-based methods in terms of ACC. Besides, the proposed method demonstrates a guarantee against forgetting reflected by 0 in BWT. Note that the proposed base model is much smaller in size compared to the other methods. Moreover, comparing to GEM Lopez-Paz & Ranzato (2017) and VCL Nguyen et al. (2018), our base network adopts far smaller memory for storing merely filter atoms for 3 layers, each of them has a hundred of parameters only.
  Table D: Results on 5-Split MNIST.
  Method ACC% BWT% Arch. Size Memory(MB) (MB)
  EWC (Kirkpatrick et al., 2017a) 95.78± 0.35 -4.20± 0.21 1.1 - HAT (Serra et al., 2018) 99.59± 0.01 0.00± 0.04 1.1 -
  UCB (Ebrahimi et al., 2019) 99.63± 0.02 0.00± 0.00 2.2 - VCL (Nguyen et al., 2018) 95.97± 1.03 -4.62± 1.28 1.1 -
  GEM (Lopez-Paz & Ranzato, 2017) 94.34± 0.82 -2.01± 0.05 6.5 0.63 VCL-C (Nguyen et al., 2018) 93.6± 0.2 -3.10± 0.20 1.7 0.63 ACL (Ebrahimi et al., 2020) 99.76± 0.03 0.01± 0.01 1.6 - MIR (Aljundi et al., 2019a) 87.60± 0.70 7.00± 0.90 5.9 0.71 GSS (Aljundi et al., 2019b) 84.80± 1.80 - 3.7 0.58
  Ours (Base) 99.84± 0.05 0.00± 0.00 0.24 0.04
sections.heading: C.7 ADDITIONAL CLASS-INCREMENTAL RESULTS
sections.text: C.7.1 ADDITIONAL RESULTS ON 10-Split CIFAR100
  We provide results 10-Split CIFAR100 under class-incremental setting with the ResNet-18 (He et al., 2016) in Tab. E and Fig. B. Our method achieves the best result with a large margin.
  Table E: Class-incremental results on 10-Split CIFAR100.
  Method Acc. (avg.) Acc. (last)
  LwF (Li & Hoiem, 2017) 44.5 23.9 EWC (Kirkpatrick et al., 2017a) 36.2 16.4 SI (Zenke et al., 2017a) 37.8 23.3 MAS (Aljundi et al., 2018b) 33.4 15.4 RWalk (Chaudhry et al., 2018b) 35.2 17.9 DMC (Zhang et al., 2020) 57.1 36.2
  Ours (Base) 58.6 40.57 w/ E = 2 62.4 46.37 w/ E = 3 62.9 46.50
  Figure B: Class incremental learning results on 10-Split CIFAR100.
  C.7.2 ADDITIONAL RESULTS ON 10-Split ImageNet-Subset
  We further validate our method on a large-scale image classification dataset, ImageNet-Subset, which is a subset of the ImageNet dataset that contains the first 100 classes. As in Masanaet et al. (2020), we split 100 classes into 10 tasks with 10 classes per task. We adopt the ResNet-18 as the architecture and substitute the convolutional layers with our decomposed version. Training details are provided in Appendix C.3.
  Table F: Class-incremental results on 10-Split ImageNet-Subset.
  Method Memory Acc. (last)
  iCaRL (Rebuffi et al., 2017)
  301.4 MB 43.6 EEIL (Castro et al., 2018) 36.6 BiC (Wu et al., 2019) 45.6 IL2M (Belouadah & Popescu, 2019) 38.2
  Ours (Base) 7.3 MB 43.9 w/ Ew = 2 7.7 MB 46.3
  We benchmark our method with existing methods that store 2000 exemplars in the external memory. Their results are faithfully reproduced in Masanaet et al. (2020). As shown in the Tab. F, our method with intra-task ensemble achieves the best result with significantly less memory usage.
sections.heading: C.7.3 ADDITIONAL RESULTS ON CUBS AND FLOWERS
sections.text: We further validate our methods on CUBS (Welinder et al., 2010) and Flowers (Nilsback & Zisserman, 2008) datasets. Following Yu et al. (2020), we use the random seed (1993) to select 100/50 classes as the first task, and
  evenly split the rest 100/50 classes into 5 tasks. We adopt ResNet-18 pretrained on ImageNet with substituted decomposed convolutional layers. Training details are illustrated in Appendix C.3.
  Table G: Average incremental accuracy on CUBS and Flowers datsets
  .
  Method CUBS Flowers
  LwF-E∗ (Li & Hoiem, 2017) 69.8 87.2 EWC-E∗ (Kirkpatrick et al., 2017a) 69.7 85.9 MAS-E∗ (Aljundi et al., 2018b) 68.5 84.7 SDC (Yu et al., 2020) 70.0 86.8
  Ours (Base) 69.3 86.9 w/ Ew = 2 72.1 89.3
  Under the class-incremental setting, we compare the average incremental accuracy with other benchmarks. On both datasets, as shown in the table above, our method produces the best results.
sections.heading: C.8 ABLATION STUDY ON DISTANCE METRICS IN INTER-TASK ENSEMBLE
sections.text: We present an ablation study of different distance metrics including the Grassmann distance, L2 norm, and cosine distance for inter-task ensemble. As shown in Tab. H, inter-task ensemble with Grassmann distance achieves the best result.
  Table H: Ablation study on different metrics in inter-task ensemble.
  Method CIFAR-100 miniImageNet
  Ours (Base) 79.13 66.01
  w/ Ec = 3 (L2 norm) 80.35 67.13 w/ Ec = 3 (cosine) 80.11 66.82 w/ Ec = 3 (Grassmann) 80.75 67.84
  C.9 INFERENCE TIME COMPARISONS
  On the CIFAR-100 dataset, we compare inference time on all classes with other methods. All methods are tested on a single RTX 2080ti GPU under the class-incremental setting. Our method has comparable inference time to other methods, as shown in the Tab. I.
sections.heading: C.10 ABLATION STUDY OF SENSITIVITY OF FIRST TASK.
sections.text: The continual learning literature mostly sample tasks from a single dataset [4,5,6]. In this commonly adopted scenario, we observe that our model is not sensitive to the selection of the first task. we provide an ablation study on 20-Split CIFAR-100. As suggested, we perform experiments with 20 different selections of the first task and calculate the statistics of results of 20 runs. Our base method achieves ACC = 79.41± 0.92, which shows a small variance.
sections.heading: C.11 MEMORY CALCULATION
sections.heading: C.11.1 CLASS-INCREMENTAL LEARNING
sections.text: We calculate the external memory usage for exemplar-based methods with 2000 exemplars, e.g., Rebuffi et al. (2017), and our method with atom memory.
  • 2000 exemplars: 2000 ∗ (32 ∗ 32 ∗ 3 + 1)/1e6 = 6.15 MB. • Our (Base): (31 ∗ 1 ∗ 12 ∗ 9 + 64 ∗ 10) ∗ 4/1e6 ∗ 9 = 0.14 MB. • Our (w/ Ew = 2): (31 ∗ 2 ∗ 12 ∗ 9 + 64 ∗ 10 ∗ 2) ∗ 4/1e6 ∗ 9 = 0.29 MB.
sections.heading: C.11.2 TASK-INCREMENTAL LEARNING
sections.text: We calculate external memory usage for our method in 20-Split CIFAR100 and 20-Split miniImageNet below.
  • Our (Base): (5 ∗ 12 ∗ 9 + 256 ∗ 5)/1e6 ∗ 4 ∗ 19 = 0.14 MB • Our (w/ Ec = 3): (5 ∗ 12 ∗ 9 + 256 ∗ 5 + 256 ∗ 5 ∗ 3)/1e6 ∗ 4 ∗ 19 = 0.43 MB • Our (w/ Ew = 2): (5 ∗ 12 ∗ 9 ∗ 2 + 256 ∗ 5 ∗ 2)/1e6 ∗ 4 ∗ 19 = 0.28 MB • Our (w/ both): (5 ∗ 12 ∗ 9 ∗ 2 + 256 ∗ 5 ∗ 2 + 256 ∗ 5 ∗ 2 ∗ 3)/1e6 ∗ 4 ∗ 19 = 0.86 MB