summaries: This paper introduces a model for continual learning based on the decomposition of linear filters into low-rank components, called atoms. Specifically, the authors decompose convolutional filters shaped (c,c',k,k) into two components: i) alpha, shaped (c,c',m) and D, shaped (m,k,k). The former is learned on the first task and then frozen, whereas for D every task has its own and they are do not conflict during optimization. On top of that, the authors envision two different ensembling schemes that improve performances. i) First, in task-incremental settings, they retrieve atoms from the task of interest and from similar tasks as well (based on SVD decomposition of D matrices and Grassman distance) and ensemble them. ii) Furthermore, in class-incremental settings, they explicitly setup multiple atoms per task, building a task ensemble. During inference, all ensembles are queried and their predictive variance is used to "discover" the relevant task, for which a prediction is carried out. Experiments are carried out on 3 datasets in both task-incremental and class-incremental learning settings.

summaries: The paper proposes a continual learning algorithm that enforces the convolutional filter in each layer to a low-rank filter subspace defined by a small set of filter atoms. For each task, each convolutional layer is defined by a new filter subspace but subspace coefficients are shared among the tasks. The algorithm is validated on multiple benchmark datasets.

summaries: The paper, motivated by the task subspace modeling literature, enforced a low-rank filter structure to each CNN layer across time in continual learning. It not only ensures that the knowledge of the past tasks is not lost but also saves a lot of computing memory. Meanwhile, the paper proposes novel intra-task ensembles and inter-task ensembles for class-incremental settings and task-incremental settings, respectively.

summaries: This paper tackles the continual learning via enforcing a low-rank filter structure to each CNN layer.  They first perform atom-coefficient filter decomposition and then learn each task with a new filter subspace, so that the method only needs to save the new filters for each task. The contribution of this paper includes the low-rank filter scheme and the designed intra-task and inter-task model ensemble performing on the filters. The proposed method also achieves SOTA performance on several datasets with tiny size of model memory.

abstractText: Continual learning is widely studied in recent years to resolve the catastrophic forgetting of deep neural networks. In this paper, we first enforce a low-rank filter subspace by decomposing convolutional filters within each network layer over a small set of filter atoms. Then, we perform continual learning with filter atom swapping. In other words, we learn for each task a new filter subspace for each convolutional layer, i.e., hundreds of parameters as filter atoms, but keep subspace coefficients shared across tasks. By maintaining a small footprint memory of filter atoms, we can easily archive models for past tasks to avoid forgetting. The effectiveness of this simple scheme for continual learning is illustrated both empirically and theoretically. The proposed atom swapping framework further enables flexible and efficient model ensemble with members selected within task or across tasks to improve the performance in different continual learning settings. The proposed method can be applied to a wide range of optimization schemes and convolutional network structures. Being validated on multiple benchmark datasets, the proposed method outperforms the state-of-the-art methods in both accuracy and scalability.

authors:
P-A Absil
Robert Mahony
Rodolphe Sepulchre

title: Riemannian geometry of grassmann manifolds with a view on algorithmic computation

venue: Acta Applicandae Mathematica,

authors:
Alessandro Achille
Michael Lam
Rahul Tewari
Avinash Ravichandran
Subhransu Maji
Charless C Fowlkes
Stefano Soatto
Pietro Perona

title: Task2vec: Task embedding for meta-learning

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,

authors:
Rahaf Aljundi
Francesca Babiloni
Mohamed Elhoseiny
Marcus Rohrbach
Tinne Tuytelaars

title: Memory aware synapses: Learning what (not) to forget

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

authors:
Rahaf Aljundi
Eugene Belilovsky
Tinne Tuytelaars
Laurent Charlin
Massimo Caccia
Min Lin
Lucas Page-Caccia

title: Online continual learning with maximal interfered retrieval

authors:
Rahaf Aljundi
Min Lin
Baptiste Goujaud
Yoshua Bengio

title: Gradient based sample selection for online continual learning

authors:
Eden Belouadah
Adrian Popescu

title: Il2m: Class incremental learning with dual memory

authors:
Francisco M Castro
Manuel J Marín-Jiménez
Nicolás Guil
Cordelia Schmid
Karteek Alahari

title: End-to-end incremental learning

venue: In Proceedings of the European conference on computer vision (ECCV),

authors:
Arslan Chaudhry
Puneet K Dokania
Thalaiyasingam Ajanthan
Philip HS Torr

title: Riemannian walk for incremental learning: Understanding forgetting and intransigence

authors:
Arslan Chaudhry
Marc’Aurelio Ranzato
Marcus Rohrbach
Mohamed Elhoseiny

title: Efficient lifelong learning with a-gem

venue: In International Conference on Learning Representations,

authors:
Arslan Chaudhry
Marcus Rohrbach
Mohamed Elhoseiny
Thalaiyasingam Ajanthan
Puneet K Dokania
Philip HS Torr
M Ranzato

title: Continual learning with tiny episodic memories. 2019a

authors:
Arslan Chaudhry
Marcus Rohrbach
Mohamed Elhoseiny
Thalaiyasingam Ajanthan
Puneet K Dokania
Philip HS Torr
Marc’Aurelio Ranzato

title: On tiny episodic memories in continual learning

venue: In Advances in Neural Information Processing Systems,

authors:
Matthias De Lange
Rahaf Aljundi
Marc Masana
Sarah Parisot
Xu Jia
Ales Leonardis
Gregory Slabaugh
Tinne Tuytelaars

title: Continual learning: A comparative study on how to defy forgetting in classification

venue: tasks. arXiv preprint arXiv:1909.08383,

authors:
Matthias Delange
Rahaf Aljundi
Marc Masana
Sarah Parisot
Xu Jia
Ales Leonardis
Greg Slabaugh
Tinne Tuytelaars

title: A continual learning survey: Defying forgetting in classification tasks

venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,

authors:
Prithviraj Dhar
Rajat Vikram Singh
Kuan-Chuan Peng
Ziyan Wu
Rama Chellappa

title: Learning without memorizing

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Simon Shaolei Du
Wei Hu
Sham M Kakade
Jason D Lee
Qi Lei

title: Few-shot learning via learning the representation, provably

authors:
Sayna Ebrahimi
Mohamed Elhoseiny
Trevor Darrell
Marcus Rohrbach

title: Uncertainty-guided continual learning in bayesian neural networks

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,

authors:
Sayna Ebrahimi
Franziska Meier
Roberto Calandra
Trevor Darrell
Marcus Rohrbach

title: Adversarial continual learning

venue: In Computer Vision–ECCV 2020: 16th European Conference,

authors:
An Evgeniou
Massimiliano Pontil

title: Multi-task feature learning

venue: Advances in neural information processing systems,

title: Toeplitz and circulant matrices: A review

authors:
Raia Hadsell
Dushyant Rao
Andrei A Rusu
Razvan Pascanu

title: Embracing change: Continual learning in deep neural networks

venue: Trends in Cognitive Sciences,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Saihui Hou
Xinyu Pan
Chen Change Loy
Zilei Wang
Dahua Lin

title: Learning a unified classifier incrementally via rebalancing

authors:
Ghassen Jerfel
Erin Grant
Thomas L Griffiths
Katherine Heller

title: Reconciling meta-learning and continual learning with online mixtures of tasks

authors:
James Kirkpatrick
Razvan Pascanu
Neil Rabinowitz
Joel Veness
Guillaume Desjardins
Andrei A Rusu
Kieran Milan
John Quan
Tiago Ramalho
Agnieszka Grabska-Barwinska

title: Overcoming catastrophic forgetting in neural networks

venue: Proceedings of the national academy of sciences,

authors:
Soheil Kolouri
Nicholas Ketz
Xinyun Zou
Jeffrey Krichmar
Praveen Pilly

title: Attention-based structuralplasticity

venue: arXiv preprint arXiv:1903.06070,

authors:
Simon Kornblith
Mohammad Norouzi
Honglak Lee
Geoffrey Hinton

title: Similarity of neural network representations revisited

venue: In International Conference on Machine Learning,

authors:
Alex Krizhevsky
Geoffrey Hinton

title: Learning multiple layers of features from tiny images

authors:
Abhishek Kumar
Hal Daume III

title: Learning task grouping and overlap in multi-task learning

venue: International Conference on Machine Learning,

authors:
Balaji Lakshminarayanan
Alexander Pritzel
Charles Blundell

title: Simple and scalable predictive uncertainty estimation using deep ensembles

authors:
Yann LeCun
Léon Bottou
Yoshua Bengio
Patrick Haffner

title: Gradient-based learning applied to document recognition

authors:
Sang-Woo Lee
Jin-Hwa Kim
Jaehyun Jun
Jung-Woo Ha
Byoung-Tak Zhang

title: Overcoming catastrophic forgetting by incremental moment matching

venue: In Proceedings of the 31st International Conference on Neural Information Processing Systems,

authors:
Xilai Li
Yingbo Zhou
Tianfu Wu
Richard Socher
Caiming Xiong

title: Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting

authors:
Zhizhong Li
Derek Hoiem

title: Learning without forgetting

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
David Lopez-Paz
Marc’Aurelio Ranzato

title: Gradient episodic memory for continual learning

authors:
Arun Mallya
Svetlana Lazebnik

title: Packnet: Adding multiple tasks to a single network by iterative pruning

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Nicolas Y Masse
Gregory D Grant
David J Freedman

title: Alleviating catastrophic forgetting using contextdependent gating and synaptic stabilization

venue: Proceedings of the National Academy of Sciences,

authors:
Andreas Maurer
Massi Pontil
Bernardino Romera-Paredes

title: Sparse coding for multitask and transfer learning

venue: In International conference on machine learning,

authors:
John Milnor
James D

title: Stasheff. Characteristic Classes.(AM-76)

venue: Volume 76,

authors:
Ari Morcos
Maithra Raghu
Samy Bengio

title: Insights on representational similarity in neural networks with canonical correlation

authors:
Cuong V Nguyen
Yingzhen Li
Thang D Bui
Richard E Turner

title: Variational continual learning

authors:
Maria-Elena Nilsback
Andrew Zisserman

title: Automated flower classification over a large number of classes

venue: Sixth Indian Conference on Computer Vision, Graphics & Image Processing,

authors:
German I Parisi
Ronald Kemker
Jose L Part
Christopher Kanan
Stefan Wermter

title: Continual lifelong learning with neural networks: A review

venue: Neural Networks,

authors:
Ameya Prabhu
Philip HS Torr
Puneet K Dokania

title: Gdumb: A simple approach that questions our progress in continual learning

venue: In European conference on computer vision,

authors:
Qiang Qiu
Xiuyuan Cheng
Guillermo Sapiro

title: Dcfnet: Deep neural network with decomposed convolutional filters

authors:
Maithra Raghu
Justin Gilmer
Jason Yosinski
Jascha Sohl-Dickstein

title: Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability

authors:
Sylvestre-Alvise Rebuffi
Alexander Kolesnikov
Georg Sperl
Christoph H Lampert

title: icarl: Incremental classifier and representation learning

venue: In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition,

authors:
Matthew Riemer
Ignacio Cases
Robert Ajemian
Miao Liu
Irina Rish
Yuhai Tu
Gerald Tesauro

title: Learning to learn without forgetting by maximizing transfer and minimizing interference

authors:
Amanda Rios
Laurent Itti

title: Closed-loop gan for continual learning

venue: International Joint Conference on Artificial Intelligence,

authors:
Hippolyt Ritter
Aleksandar Botev
David Barber

title: Online structured laplace approximations for overcoming catastrophic forgetting

venue: In NeurIPS,

title: Catastrophic forgetting, rehearsal and pseudorehearsal

venue: Connection Science,

authors:
Bernardino Romera-Paredes
Hane Aung
Nadia Bianchi-Berthouze
Massimiliano Pontil

title: Multilinear multitask learning

authors:
Syed Shakib Sarwar
Aayush Ankit
Kaushik Roy

title: Incremental learning in deep convolutional neural networks using partial network sharing

venue: IEEE Access,

authors:
Jonathan Schwarz
Wojciech Czarnecki
Jelena Luketina
Agnieszka Grabska-Barwinska
Yee Whye Teh
Razvan Pascanu
Raia Hadsell

title: Progress & compress: A scalable framework for continual learning

authors:
Joan Serra
Didac Suris
Marius Miron
Alexandros Karatzoglou

title: Overcoming catastrophic forgetting with hard attention to the task

authors:
Hanul Shin
Jung Kwon Lee
Jaehong Kim
Jiwon Kim

title: Continual learning with deep generative replay

authors:
Binh Tang
David S Matteson

title: Graph-based continual learning

authors:
Michalis K Titsias
Jonathan Schwarz
Alexander G de G Matthews
Razvan Pascanu
Yee Whye Teh

title: Functional regularisation for continual learning with gaussian processes

authors:
Oriol Vinyals
Charles Blundell
Timothy Lillicrap
Daan Wierstra

title: Matching networks for one shot learning

authors:
P. Welinder
S. Branson
T. Mita
C. Wah
F. Schroff
S. Belongie
P. Perona

title: Caltech-UCSD Birds 200

venue: Technical Report CNS-TR-2010-001, California Institute of Technology,

authors:
Mitchell Wortsman
Vivek Ramanujan
Rosanne Liu
Aniruddha Kembhavi
Mohammad Rastegari
Jason Yosinski
Ali Farhadi

title: Supermasks in superposition

authors:
Yue Wu
Yinpeng Chen
Lijuan Wang
Yuancheng Ye
Zicheng Liu
Yandong Guo
Yun Fu

title: Large scale incremental learning

authors:
Jaehong Yoon
Eunho Yang
Jeongtae Lee
Sung Ju Hwang

title: Lifelong learning with dynamically expandable networks

authors:
Jaehong Yoon
Saehoon Kim
Eunho Yang
Sung Ju Hwang

title: Scalable and order-robust continual learning with additive parameter decomposition

authors:
Lu Yu
Bartlomiej Twardowski
Xialei Liu
Luis Herranz
Kai Wang
Yongmei Cheng
Shangling Jui
Joost van de Weijer

title: Semantic drift compensation for class-incremental learning

authors:
Amir R Zamir
Alexander Sax
William Shen
Leonidas J Guibas
Jitendra Malik
Silvio Savarese

title: Taskonomy: Disentangling task transfer learning

authors:
Friedemann Zenke
Ben Poole
Surya Ganguli

title: Continual learning through synaptic intelligence

authors:
Junting Zhang
Jie Zhang
Shalini Ghosh
Dawei Li
Serafettin Tasci
Larry Heck
Heming Zhang
C-C Jay Kuo

title: Class-incremental learning via deep model consolidation

venue: In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,

authors:
Yu Zhang
Qiang Yang

title: A survey on multi-task learning

venue: IEEE Transactions on Knowledge and Data Engineering,

heading: 1 INTRODUCTION

text:
Humans keep acquiring new concepts without forgetting crucial ones in the past. To endow intelligent agents with the same ability of long-term knowledge accumulation, continual learning (CL) has been intensively studied in recent years. In continual learning, an agent learns from a sequence of tasks, with the goal of gaining knowledge of each new task while preserving the capacity for resolving the old ones, therefore to avoid catastrophic forgetting. The recent advances of CL mainly follow several directions. One popular category among them is to maintain an external memory of the original images (Robins, 1995; Rebuffi et al., 2017), synthesized images (Shin et al., 2017), or parameter gradients (Lopez-Paz & Ranzato, 2017) for archiving the past. These memory-based methods often suffer from heavy memory footprints, while still forgetting about the previous tasks to some extent.
Motivated by the literature on subspace modeling of tasks (Evgeniou & Pontil, 2007; Maurer et al., 2013; Zhang & Yang, 2021; Romera-Paredes et al., 2013; Kumar & Daume III, 2012), in this paper, we propose to learn for each task a new filter subspace for each convolutional layer, i.e., hundreds of parameters as filter atoms, but keep subspace coefficients shared across tasks. In other words, in a CNN, we enforce a low-rank filter subspace by decomposing convolutional filters within each network layer over a small set of filter atoms. Then, we perform continual learning by simply swapping filter atoms for each task. The effectiveness of our approach is empirically validated and further explained theoretically with an excess risk bound analysis.
With the proposed approach, we can faithfully remember the past by only maintaining a small footprint atom memory to archive task-specific filter atoms. Any previously learned CNN models can now be exactly recovered by multiplying the task-shared coefficients with the task-specific atoms, which can be retrieved efficiently from the atom memory. Thus, the introduced filter atom memory permits historical knowledge to be recalled with a guarantee against forgetting. While comparing with state-of-the-art memory-based CL methods, our approach requires only storing for each task
∗This work is supported by DARPA TAMI program under No. HR00112190038.
some tiny size of filter atoms, which in total are typically much smaller than the size of exemplars in most memory-based methods (Rebuffi et al., 2017; Prabhu et al., 2020), and therefore potentially supports continual learning on a large scale.
Our atom swapping continual learning framework can effectively support both inter-task and intra-task model ensemble to further enhance performance in different continual learning settings: First, intertask ensemble utilizes the relevant past knowledge to boost the present task performance. To ensure that ensemble with past models can affect the current task positively, we only select a relevant subset of past models. With a life-long learning scenario in mind, we choose to assess task relevancy simply based on the filter subspace distance, which can be on-the-fly computed here via the Grassmann distance (Absil et al., 2004) among task-specific filter subspaces. Second, intra-task ensemble can be adopted in class-incremental setting to help task prediction with the minimal-entropy criterion. Usually, ensemble members are instantiated as independent CNNs, and their learning and inference are conducted separately for dissimilarity. However, this will lead to significant increase in training and inference time and memory usage. We address this problem by creating within a task multiple virtual members in a single CNN model by simply maintaining several groups of filter atoms in each layer. In this way, different intra-task members are integrated into a single network, and its learning and inference can be conducted efficiently with group convolution.
We validate our simple yet effective approach on several continual learning benchmarks such as MNIST, CIFAR100, and miniImageNet using both class-incremental and task-incremental settings, and observe competitive results against state-of-the-art methods on all benchmarks with far less memory usage.
We summarize our contributions as follows,
• We learn for each task a new filter subspace for each layer, and keep subspace coefficients shared across tasks.
• We maintain a small footprint filter atom memory that can faithfully archive past knowledge with a guarantee against forgetting in a highly scalable way.
• We adopt inter-task ensemble for the present task by recalling past models based on an on-the-fly calculated task relevancy in task-incremental settings.
• We propose intra-task ensemble for class-incremental settings by creating multiple virtual members in a single CNN model through different groups of filter atoms per layer.

heading: 2 MOTIVATION

text:
We are motivated by the literature on task subspace modeling (Evgeniou & Pontil, 2007; Maurer et al., 2013; Zhang & Yang, 2021; Romera-Paredes et al., 2013; Kumar & Daume III, 2012), where it is commonly assumed that task parameters lie in a low dimensional subspace, so that tasks can be modeled as a set of latent basis tasks and their linear combinations. The latent basis tasks and the respective linear combinations are often obtained via alternative optimization by fixing one and optimizing the other (Kumar & Daume III, 2012).
In our continual learning setting, we model tasks using convolutional neural networks (CNNs). Following (Qiu et al., 2018), we decompose a convolutional filter Wi ∈ Rc×c
′×k×k for the i-th layer over m filter atoms Di ∈ Rm×k×k, linearly combined by coefficient αi ∈ Rc×c
′×m, where c and c′ are the number of input and output channels, k is the spatial size of each atom. This can be written as Wi = αi ×Di. Note that this decomposition distributes a filter Wi into two imbalanced parts: αi for channel mixing with mcc′ parameters, and light-weight Di for spatial filtering with only mk2 entries. In all, we use W = α×D to denote the filters decomposition in a model with l convolutional layers, where W = {Wi}li=1,α = {αi}li=1,D = {Di}li=1 indicate all filters, coefficients, and atoms respectively.
It is easy to observe that, within each CNN layer, we can borrow from the task subspace modeling methodology by creating a set of latent basis tasks through filter atoms Di and their linear combinations as atom coefficients αi. The linear combination coefficients αi are learned on the first task jointly with the first group of atoms, and then atoms for subsequent tasks are alternatively optimized by fixing αi.
As illustrated in the subsequent sections, this seemingly under-fitting method not only enables an efficient way to faithfully archive past models with a guarantee against forgetting, but also supports efficient inter-task and intra-task model ensemble to further improve the performance.

heading: 3 METHODOLOGY

text:
We consider the problem of learning T tasks sequentially. Formally, we denote the data distribution that associates with the t-th task as Dt = (X t,Yt), t ∈ {1, 2, ..., T}, from which a dataset Dt = {xit,yit} Nt i=1 is sampled for training. The goal of continual learning is to minimize the statistical risk of all seen tasks given no access to data Dt from previous tasks t ≤ T (Delange et al., 2021): T∑
t=1
E(X t,Yt)[L(F t(X t; θ),Yt)], (1)
where L denotes the risk function, F t(·; θ) is the model for task t with parameter θ. Continual learning with a guarantee against forgetting can be achieved by storing learned parameters entirely in an external memoryM = {θt}Tt=1 given F t(·; θ) = F(·; θt), thus any previous model can be completely recovered by retrieving the corresponding parameters from the memory. However, such a straightforward solution based on parameter memory suffers severely on its poor scalability due to the large size of modern deep neural networks and the potentially long task sequence. We address the scalability issue and achieve guaranteed non-forgetting by decomposing convolutional filters in a CNN into task-specific filter atoms and task-shared coefficients. Then only hundreds of parameters per task need to be stored in an atom memory to guarantee non-forgetting. This proposed approach allows efficient inter-task and intra-task ensemble to further boost performance.

heading: 3.1 A SCALABLE APPROACH AGAINST FORGETTING

text:
In CNNs, catastrophic forgetting occurs when a model learned from a sequence of past tasks is updated in favor of the current task, resulting in significant performance degradation. A straightforward solution, as mentioned before, is to archive Wt in an external memoryM = {Wt}Tt=1, and the representation space of any previous model can be faithfully recalled by memory retrieval. However, as deep CNNs contain great amount of parameters in Wt, this simple solution scales poorly with the number of tasks T . On the other hand, storing part of parameters, or small subsets of data for parameter flashback cannot avoid forgetting completely (Sarwar et al., 2019; Yoon et al., 2018).
This dilemma can be resolved with the proposed filter atom decomposition, as shown in Fig. 1. With the filter decomposition described in Sec. 2, by storing task-specific filter atoms Dt into memory, and enforcing a task-shared coefficients α, the model archives the entire knowledge for each time point.
We refer to the memory for storing atoms as the atom memory,MD = {Dt}Tt=1,Dt ∈ Rm×k×k, with each Dt learned in the t-th task with empirical risk minimization,
arg min Dt Nt∑ i=1 L(F(xti;α,Dt),yti). (2)
The task-shared coefficients α are learned on the first task jointly with the first group of atoms,
arg min D1,α N1∑ i=1 L(F(x1i ;α,D1),y1i ). (3)
In this way, we can guarantee that the statistical risk for a previous task t at any time point remains the same as we can recall faithfully the past model by multiplying the stored atoms with the task-shared coefficients,
E(X t,Yt)[L(F t(X t; θ),Yt)] = E(X t,Yt)[L(F(X t;α×Dt),Yt)]. (4)
Atom memory scalability. The proposed atom memory stores a group of atoms per task, which is scalable with increasing number of tasks. Formally, consider a l-layer CNN model with associated filters W = {Wi}li=1. As mentioned in Sec. 2, each filter can be decomposed as Wi = αi ×Di. The group of atoms for task t can then be denoted as Dt = {Dti}li=1, which requires a size of lmk2 in storing parameters per task. This typically introduces only a few hundred of parameters for each task to be stored in the atom memory, which potentially supports continual learning on a large scale. Details for scalability comparison are shown in Sec. 5.2.1.
Analysis of excess risk bound. With task-shared coefficients α, the model for each new task may seemingly expect some degree of underfitting. However, as demonstrated in Section 5, we still observe superior results over the state-of-the-art continual learning methods on all benchmarks we have evaluated. To understand this, we here theoretically analyze the excess risk bound for new tasks in the continual learning setting. For each tasks t ∈ {1, 2, ..., T}, model F t(·; θ) consists of representation function φt and prediction function wt (wt ∈ Rk). The representation function φt maps an input x to feature space Z ⊆ Rk. For the analysis purpose, we assume φt is just one convolution layer, which can be decomposed as φt = α×Dt. Therefore, using the training samples from task t, we can solve the following optimization problem from Eq. (2):
min 1
2N t ‖yt − (α×Dt(xt))+wt‖2 +
λ 2 ‖α×Dt‖2F + λ 2 ‖wt‖2F , (5)
where we use the mean square error as our loss, (·)+ is ReLU activation (z)+ = max{0, z}. The shared coefficients α are learned on the first task jointly with the first group of atoms. Similar to (Du et al., 2020), we also assume that there is a ground-truth optimal representation function φt,∗ and prediction function wt,∗ for task t.
Assumption 1 (subgaussian input). There exists ρ > 0 such that, for all t ∈ {1, 2, ..., T}, the random vector x̄ ∼ p̄t is ρ2-subgaussian. The p̄t is the distribution of samples in task t. Assumption 2 (oracle network). Assume for task t ∈ {1, 2, ..., T} that yt = (α∗×Dt,∗(xt))+wt,∗+ zt is generated by an oracle network with parameters α∗, Dt,∗, and wt,∗. Noise term zt ∼ N (0, σ2I). Excess risk bound. We can bound the excess risk of our learned model on the task t, i.e., how much our learned model (α̂, D̂t, ŵt) performs worse than the optimal model (α∗,Dt,∗,wt,∗) on the task t as follows:
ER(α̂, D̂t, ŵt) = LDt(α̂, D̂ t, ŵt)− LDt(α∗,Dt,∗,wt,∗) ≤ σR̄ · Õ( √ Tr(Σ) + √ ‖Σ‖2√
N t ) + ρ4R̄2 · Õ(Tr(Σ) + ‖Σ‖2 N t )
where R̄ = 12‖α ∗ ×Dt,∗‖2F + 12‖w t,∗‖2F , Σ = Ex∼p[xxᵀ], N t is the number of training samples. Õ is the big O notation, and LDt is the expected loss with the data distribution Dt. The detailed analysis is provided in the Appendix A.

heading: 3.2 INTER-TASK MODEL ENSEMBLE

text:
The atom memory not only serves as an efficient way to archive past models with a guarantee against forgetting, but also enables efficient recall of past models for model ensemble to improve the performance in present task. As in Breiman (2001); Lakshminarayanan et al. (2017), ensemble performance increases with the independent level of ensemble members. Motivated by this, model ensemble over time for the current task at time r has a natural advantage that each ensemble member F t (t ∈ {1, 2, ..., r}) is learned from a different data distributionDt. However, most of other methods are not affordable to perform model ensemble across tasks since they often lack an effective and efficient way to recall past models. As illustrated in Fig. 2, with the atom memory, our method can faithfully and rapidly recall past models F t = F(·;α×Dt) by simply fetching atoms Dt from the atom memory; and then perform inter-task model ensemble by constructing a uniformly-weighted mixture model and combine the predictions as (Lakshminarayanan et al., 2017),
Frc−ens(x) = 1 |Sr|+ 1 ∑
s∈Sr∪{r}
pF (y|x, θs) = 1
Ec + 1 ∑ s∈Sr∪{r} F(x;α×Ds), (6)
where Sr denotes the index set of previous tasks used for ensemble, and Ec = |Sr|. For classification problem, it corresponds to averaging the predictive probabilities.
In continual learning, not all past models can bring positive effects on the current task. According to (Breiman, 2001; Lakshminarayanan et al., 2017), only the ones that have enough strengths on the current r-th task can help enhance the performance as weak learners (Breiman, 2001), which can be selected based on task relevancy. Our assumption is that the more a past task t resembles the current r, the better performance will F t achieves on the present task. The problem then transforms to evaluating the model similarity effectively and efficiently. Note that to ensure the proposed ensemble method to be scalable across a very long historical task sequence T , highly efficient task relevancy assessment is indispensable here.
Assessing task relevancy by filter subspace distance. Task relevancy assessments proposed in (Achille et al., 2019; Zamir et al., 2018) work at the cost of heavy computation, which prevents their efficient applications in continual learning. While measuring the similarities among the learned models can be a straightforward proxy to the measurement of task relevancy, the widely studied methods, e.g., canonical correlation analysis (CCA) (Raghu et al., 2017; Morcos et al., 2018) and centered kernel alignment (CKA) (Kornblith et al., 2019), still introduce considerable computational cost while performing evaluations in the representation space. Directly performing relevancy measurements by calculating the distance of filters works at highly desirable efficiency, yet can perform poorly without the costly semantic alignments over channels (Raghu et al., 2017). In our approach, thanks to the task-shared coefficients acting as structural regularizations, we show that model similarity measurements can now be efficiently evaluated through direct filter similarity measurements. And as we model filters using task-specific filter subspaces with coefficients shared, model similarity can be further reduced to assessing filter subspace distance via the Grassmann distance (Absil et al., 2004).
Formally, we characterize the filter subspace of the current model Vr and a past model Vt as: Vr = Span{Br1, ...,BrM}, Vt = Span{Bt1, ...,BtM}, Vr,Vt ⊂ RL, (7)
where L = k2 is the dimension of kernel space, B = {Bj}Mj=1 are M (m > M ) linear independent vectors serve as the bases of the filter subspace. We obtain B by performing singular value decomposition (SVD) to atoms D = UΣV T ∈ RL×m and select the first M columns from U that correspond to the top-M singular values. By definition, Vr and Vt are points of the Grassmann Manifold (Milnor & Stasheff, 2016), i.e., Vr,Vt ∈ Grass(M,L) , {M dimensional subspaces of RL}. Then, the Grassmann distance between Vr and Vt is defined as,
dM (Vr,Vt) = ( M∑ j θ2j ) 1/2, (8)
where θj is the j-th principle angle, which can be calculated by,
θj = arccos(σj), with (Br)TBt = UΣV T , σj = Σjj . (9)
The proposed task relevancy measurement requires merely a SVD to matrices with dimensions lower than L. In practice, k = 3 so that L = 9, indicating that computation of dM is low. With the proposed efficient task relevancy measurement adopted in the last convolutional layer, we select the most Ec relevant models from previous r − 1 ones that support the current task. We provide the correlation analysis of task similarities measured with CCA (Raghu et al., 2017) and the ones assessed by the proposed subspace distance in Appendix. C.5.
Although it is known that the ensemble result increases in the ensemble number Ec (Lakshminarayanan et al., 2017), it does not hold in our setting based on our empirical observation. In fact, determining the Ec is a trade-off between the amount and the relevancy of past knowledge. We thus empirically select the ensemble number Ec, which is illustrated in Sec. 5.2.1. The selected members are further fine-tuned with new classification heads to the current task.

heading: 3.3 INTRA-TASK MODEL ENSEMBLE

text:
Our filter decomposition not only allows ensemble with past members, but also permits an efficient way to create ensemble members within a task. In the regular deep ensemble scenario (Lakshminarayanan et al., 2017), different members are instantiated as multiple CNNs, and need to be learned separately to ensure independence among members. Plus, obtaining ensemble results also requires inferences with multiple CNNs. This introduces significant cost in both time and memory, making it inappropriate in continual learning settings. The proposed atom decomposition allows a new way of parameterization of ensemble members within a task to improve performance while substantially reducing the training and testing time cost. Formally, given task t, the intra-task ensemble model F tw−ens is composed by {F̂ t,1, ..., F̂ t,Ew}, where Ew is the number of models. Rather than instantiating them as different CNN models, we reparameterize them with member-specific atoms {D̂t,1, ..., D̂t,Ew}, and member-shared coefficient αt, as shown in Fig. 3. In this way, we create multiple virtual members using a single CNN model by simply maintaining different groups of filter atoms in each convolutional layer. The forward pass of F tw−ens can then be conducted by group convolution,
F tw−ens(xt) = 1
Ew Ew∑ i=1 F̂ t,i(xt) = F tw−ens(x̂t, Ŵ tw−ens), (10)
where x̂t ∈ Rn×(c×Ew)×h×w is the input repeated by Ew times, Ŵ tw−ens = [α × D̂t,1 | ... | α × D̂t,Ew ] ∈ Rc×(c′×Ew)×k×k (| denotes concatenation) is the filter for group convolution that is concatenated from per-member reconstructed filters. To enforce the independence of different members, we initialize member-specific atoms separately before training.
With diverse predictions from different virtual members, our intra-task ensemble can directly boost the performance in task-incremental settings. Furthermore, our intra-task model ensemble makes the model to better distinguish data from out-of-task distribution, i.e., xp(p 6= t), and shows high entropy in its predictive distribution which is particularly useful in class-incremental (CI) setting. Note that in CI we perform task prediction first, select a specific group of atoms, and then perform classification within task. Task id is selected based on minimal-entropy criterion on predictive distributions. Thus, intra-task ensemble can enhance the accuracy of task prediction, and then the overall CI performance1.
1Code is available at: https://github.com/ZichenMiao/CL_Atom_Swapping

heading: 4 EXPERIMENTAL SETUP

text:
Datasets. We first validate our method on the Class-Incremental (CI) setting with CIFAR100 and ImageNet-Subset , which contains 100 classes selected from ImageNet (with random seed 1993). For each dataset, half of classes are used for model learning in the 0-th task. Then the remaining classes are equally split into N tasks, where N can be 5 or 10. Note that in the CI setting, task information is not provided during testing. Then, we report the performance of our method under the Task-Incremental (TI) setting. We validate our approach on 5-Split MNIST (LeCun et al., 1998), 20-Split CIFAR100 (Krizhevsky et al., 2009), and 20-Split miniImageNet (Vinyals et al., 2016). The 5-Split MNIST uniformly splits the original 10 classes of 0-9 MNIST digits into 5 sequential tasks. 20-Split CIFAR100 and 20-Split miniImageNet are both constructed by randomly splitting 100 classes into 20 tasks with 5 classes per task. Details of each dataset are provided in the Appendix C.
Network architectures and implementation details. For the class-incremental setting, we utilize ResNet-32 for CIFAR100 and ResNet-18 for ImageNet-Subset as (Rebuffi et al., 2017). For the task-incremental setting, we adopt an AlexNet-like network. Note that we substitute all convolutional layers in both models with our decomposed version. Details of architecture are shown in the Appendix C. In terms of the proposed ensemble strategies, inter-task ensemble is only deployed in the task-incremental setting as task IDs are needed, while intra-task ensemble is utilized in both taskincremental and class-incremental settings. In inter-task ensemble, we set M = 3 for the dimension of filter subspaces. When intra-task ensemble and inter-task ensemble are adopted together, we use the member with the best results in every task for calculating task relevancy. We provide the ablation study for the inter-class and intra-class ensemble numbers Ec and Ew in Sec. 5.2.1. Training details are showed in the Appendix C.3. For forward knowledge transfer, the most recent atoms are used to initialize the atoms of the current task.
Evaluation metrics. In the class-incremental setting, we evaluate the model’s average classincremental accuracy. In the task-incremental setting, we measure the performances with ACC as the average test accuracy across all tasks. To measure the forgetting, we adopt the backward transfer, BWT, which shows how the previous tasks performance has degraded due to learning new tasks. Details of these two measurements are provided in Appendix C.4.

heading: 5 RESULTS AND DISCUSSION

text:
In this section, we start from the challenging class-incremental setting. Then we move to taskincremental setting with self-comparison experiments to validate the effectiveness of some key ingredients of our methods, and show the results of the proposed method on several real-world
datasets. In both settings, our method achieves improvements over state-of-the-art methods with significant less memory usage.

heading: 5.1 CLASS-INCREMENTAL EXPERIMENTS

text: As a more challenging setting, class-incremental (CI) learning does not provide task id during testing. As mentioned in Sec. 3.3, we handle this setting by breaking it down to a two-level task, task prediction based on minimal-entropy criterion, and then within-task classification. In CI experiments, we select the number of atoms m = 12 and the number of members for intra-task ensemble Ew = 2. We benchmark our method by comparing with many existing methods that store 2000 exemplars in the external memory. As shown in Tab. 1, our method with intra-task ensemble achieves the best results with an order of magnitude less memory usage on both CIFAR100 and ImageNet-Subset, which validates the effectiveness and scalability of our framework in the challenging CI setting.

heading: 5.2 RESULTS ON TASK-INCREMENTAL SETTINGS

text: 5.2.1 SELF COMPARISONS ON 20-Split CIFAR100 In this section, we empirically analyze inter-task ensemble, intra-task ensemble, and selection of atoms of our method in the task-incremental setting. We analyze the performance of our base model with different number of atoms m, As shown in the left of Fig. 4, m = 12 is the best choice in terms of both performance and efficiency. We then test the task relevancy assessment based on subspace distance, and its instructive effect to inter-task ensemble. As shown in Fig. 5, the past models with small Grassmann distance to the current model lead to performance improvement by model ensemble. And past models with large Grassmann distance to the current one, in fact result in degraded performances. Note that we also present an ablation study on different metrics in Appendix C.8, which shows the superior performance of Grassmann distance over L2 and cosine distance. In general, we test the ACC of model ensemble with the top-1 to top-3 relevant model as well as the most irrelevant model starting from the 5-th task. As shown in Fig. 5, model ensemble with top-3 relevant models achieves the best results, and thus we set Ec = 3 in subsequent experiments. We then explore intra-task ensemble with different members Ew = 2, 3, 4. As illustrated in the middle of Fig. 4, ensemble within task enhances the performance consistently, and we chooseEw = 2 for the best performance efficiency trade-off. We further illustrate the scalability of the proposed method on the right of Fig. 4. Compared to PNN (Rusu et al., 2016) and APD (Yoon et al., 2019), the size of our method scales much slower with the number of tasks, even with ensemble adopted.

heading: 5.2.2 COMPARISONS WITH BENCHMARKS

text: We further report our results on 20-Split miniImageNet and 20-Split CIFAR100 in Tab. 2. Comparing with regularization-based, memory-based, and expansion-based models, the proposed method achieves the best results even with the base model. Adopting inter-task ensemble with the top-3 relevant past models, along with intra-task ensemble with 2 members, our method achieves further improved results. Especially in 20-Split miniImageNet, the previous state-of-the-art method, ACL (Ebrahimi et al., 2020) achieves ACC of 62.07 with a memory in size of 8.5 MB; whereas the proposed method improves the results significantly to 67.84 with a merely 0.86 MB memory. When comparing with other memory-based methods besides ACL, the proposed method demonstrates superior scalability reflected by the much smaller memory size. We provide additional results on the standard 5-Split MNIST dataset with our base model. As shown in Tab. D, our base method outperforms both regularization-based methods and memory-based methods in terms of ACC with much smaller memory. The improvements over the state-of-the-art methods and the outstanding scalability validate the effectiveness of our method on solving real-world continual learning problems.

heading: 6 RELATED WORK

text:
Catastrophic forgetting is the central challenge of continual learning. Recent advances on continual learning is mainly driven by three main directions, which we will briefly review in terms of regularization-based, memory-based and expansion-based methods. For a more extensive overview we refer to the reviews by (De Lange et al., 2019; Parisi et al., 2019).
Regularization-based methods. (Kirkpatrick et al., 2017b; Aljundi et al., 2018a; Lee et al., 2017; Zenke et al., 2017b; Kolouri et al., 2019) determine the importance of each model’s parameter per task, which prevents the important parameters from being updated for new tasks. For example, (Kirkpatrick et al., 2017b) specify the performance of each weight with the Fisher information matrix. (Aljundi et al., 2018a), on the other hand, proposes determining parameter importance by gradient magnitude. Theses methods can be naturally explored from the lens of Bayesian optimization (Nguyen et al., 2018; Titsias et al., 2020; Schwarz et al., 2018; Ebrahimi et al., 2019; Ritter et al., 2018). Specifically, (Nguyen et al., 2018) introduces a regularization technique, inspired by variational inference, to prevent their model from forgetting. All these methods address catastrophic forgetting by adding regularization terms. As pointed out in (De Lange et al., 2019), the penalty term proposed in such methods are unable to prevent drifts in the loss landscape of previous tasks. While alleviating forgetting, the penalty also unavoidably prevents the plasticity to absorb new in- formation from future tasks learned over a long timescale (Hadsell et al., 2020).
Memory-based methods. (De Lange et al., 2019) assume it is feasible to access data from previous tasks by having a fixed-size memory or a generative model able to produces samples from previous tasks (Lopez-Paz & Ranzato, 2017; Riemer et al., 2018; Rios & Itti, 2018; Shin et al., 2017). (Rebuffi et al., 2017) introduces models augmented with fixed-size memory, which accumulates samples in the proximity of class centers. (Chaudhry et al., 2019b) proposes another memory-based model by exploiting a reservoir sampling strategy in the raw input data selection phase. Rather than storing the original samples, (Chaudhry et al., 2018a) accumulates the parameter gradients during task learning. (Shin et al., 2017) incorporate a generative model into a continual learning model to alleviate catastrophic forgetting by producing samples from previous task and retraining the model using data from previous tasks and the the current one. These methods assume an extra neural network, such as a generative model or a memory. Different from replay-based methods, which benefit from a memory to retrain their model over previous tasks, our method requires storage of tiny atoms for each previous task only, which is more scalable and do not suffer from potential forgetting caused by the inconsistent memory reply in generative-based methods.
Expansion-based methods. (Rusu et al., 2016; Yoon et al., 2018; Jerfel et al., 2019; Li et al., 2019) allocate a subset of the model parameters for each task. Model expansion can be achieved by a gating mechanism (Wortsman et al., 2020; Masse et al., 2018), or by incrementally adding new parameters to the models (Rusu et al., 2016). Incrementally learning and pruning provides another direction (Mallya & Lazebnik, 2018). Given an over-parametrized model with the ability to learn potentially many tasks, (Mallya & Lazebnik, 2018) achieves model expansion by pruning the parameters not contributing to the performance of the current task, while keeping them avail- able for future tasks. Comparing to the aforementioned methods, the proposed method provides a novel view of subspace of deep convolutional neural networks, which allows model expansion to be achieved with significantly decreased parameters, and is therefore scalable to even thousands of tasks.

heading: 7 CONCLUSION

text: In this paper, motivated by the task subspace modeling literature, we enforced a low-rank filter structure to each CNN layer across time in continual learning. By performing atom-coefficient filter decomposition, we learned for each task a new filter subspace for each layer, while keeping subspace coefficients shared across tasks. This simple method allows highly efficient model storage and retrieval using a small footprint atom memory. The proposed method provided a guarantee against forgetting, and we demonstrated further performance improvements through model ensemble. The performance was evaluated on various continual learning tasks, and the effectiveness and scalability were demonstrated by the state-of-the-art accuracy and the tiny size of model memory.

heading: APPENDIX

heading: A EXPECTED EXCESS RISK

text:
Problem setup: Suppose we have T tasks sequentially. The data distribution Dt = (X t,Yt) associates t-th task (t ∈ {1, 2, ..., T}). For each task t ∈ [T ], we assume φt : X −→ Z is a representation function, which maps input images to feature space Z ⊆ Rk. These representation functions are restricted in function class Φ, such as neural networks. There is also a predictor wt : Z −→ Y for task t ∈ [T ] mapping the feature representation to labels. Our object is to minimize empirical risk:
E(X t,Yt)[L(F t(X t; θ),Yt)].
For analysis purpose, we consider representation function φt as single convolutional layer, which can be decomposed as φt = α×Dt. With the predictor wt, the loss can be written as:
L(F t(X t; θ),Yt) = 1 2N t Nt∑ i=1 (yi − 〈wt, (α×Dt(xi))+〉)2,
where Dt(x) is the convolution operation on the input images, (·)+ is ReLU activation. As the input x ∈ Rc×h×w and atoms D ∈ Rm×k×k, where h× w are the size of input images, c is the number of input channel, and m is the number of kernels which have size of k × k. Dt(x) contains c×m convolution operations that maps input x into the latent zl ∈ Rcm×h ′×w′ . The latent zl then maps into feature space Z by α× zl.
We can treat each convolution operation as matrix product by converting the convolution kernel to a doubly blocked Toeplitz matrix D′t ∈ Rhw×h
′w′ (Gray, 2006), and converting input image x into x′ ∈ Rc×hw. Therefore, the convolution operation Dt(x) is transformed as reshaped image x′ multiple by m reformed kernels D′t. To simplify the problem, we assume consider each channel of the input images and each base of atoms, such that c = 1 and m = 1. Consider of all N t samples Xt ∈ RN t×hw with labels Y t ∈ RN t
for task t, Thus, the loss can be written as:
L(F t(X t; θ),Yt) = 1 2N t Nt∑ i=1 (yi − 〈wt, (α×Dt(xi))+〉)2
= 1 2N t ‖Y t − (αᵀXtD′t)+wᵀt‖2
The first task trains the α, D′1, and w1 at the same time, but the following tasks only optimize over Dt and wt and keep α fixed. With fixed α, we can represent the samples as X̃t = αᵀXt. By solving the object, we get (D̂t, ŵt). We also assume that there is an optimal atom Dt,∗ of representation function φt,∗ and prediction function wt,∗ for task t. We now analyze the bound for the excess risk of our learned model on the task t, i.e., how much our learned model (D̂t, ŵt) performs worse than the optimal model (Dt,∗,wt,∗) on the task t:
ER(φ̂t, ŵt) = LDt(D̂t, ŵ t)− LDt(Dt,∗,wt,∗)
= 1 2 ((X̃tD̂′ᵀt)+ŵ
ᵀt − (X̃tD′ᵀt,∗)+wᵀt,∗)2 (1)
A standard lifting of neural networks can be formulated as infinite dimension linear regression (Du et al., 2020). Define the infinite feature vector with the coordinates φ(x)b = (bᵀx)+ for every b ∈ Sd0−1. Let βt be a signed measure on Sd0−1. The inner product notation denotes integration: βᵀφ(x) = ∫ Sd0−1 φ(x)bdβb. By convert input feature x to the lifted feature vector φ(x), according to (Du et al., 2020), Equation. 1 can be bounded by: R(φ̂t, ŵt) = 1
2 (X̃tD̂′ᵀtŵᵀt − X̃tD′ᵀt,∗wᵀt,∗)2.
Let E[x̃x̃ᵀ] = Σ̃, we have 1
2 (X̃tD̂′ᵀtŵᵀt − X̃tD′ᵀt,∗wᵀt,∗)2 = 1 2 ‖Σ̃1/2D′ᵀt,∗wᵀt,∗ − Σ̃1/2D̂′ᵀtŵᵀt‖2.
With Assumption 1 and 2, based on the inequality in (Du et al., 2020), for a fixed δ > 0, we further have 1
2 ‖Σ̃1/2D′ᵀt,∗wᵀt,∗ − Σ̃1/2D̂′ᵀtŵᵀt‖2 ≤ 2ee,1R̄+ 2ic,1R̄2 + 2ee,2r + 2ic,2r2,
where 2ee,1 = 1√N1 σ(log 1 δ )3/2log(n) √ ‖Σ̃‖+ Tr(Σ̃), 2ee,2 = 1√Nt σ(log 1 δ )3/2log(n) √ ‖Σ̃‖ are estimation error and ic,1 = 2Cρ 2
√ N1
( √ Tr(Σ̃) + √ log 2
δ ‖Σ̃‖), ic,2 = Cρ 2 √ Nt
√ ‖Σ̃‖R̄log 1
δ are intrinsic dimension concen-
tration error, R̄ = 1 2 ‖α∗ ×Dt,∗‖2F + 12‖w t,∗‖2F . Therefore, suppose we have the same number of samples
in each task, with probability at least 1− δ over the samples, the expected excess risk of the learned atom and predictor on the task satisfies:
E[ER(φ̂t, ŵt)] ≤ σR̄ · Õ( √ Tr(Σ) + √ ‖Σ‖2√
N t ) + ρ4R̄2 · Õ(Tr(Σ) + ‖Σ‖2 N t )

heading: B ALGORITHM

text:
We provide the algorithm of the proposed method in Alg. 1.
Algorithm 1 Continual Learning with Filter Atom Swapping
Initialize MD ←− [ ], α = {αi}li=1 ←− α0, D1 = {D1i }li=1 ←− D1,0. for task t = 1, 2, ..., T do
if t == 1 then Optimize α,D1 according to (3). Update Atom Memory MD ←− [MD,D1]. else if t < 5 then
Initialize Dt = {Dti}li=1 ←− Dt,0, then optimize Dt according to (2). Update Atom Memory MD ←− [MD,Dt]. else Initialize Dt = {Dti}li=1 ←− Dt,0, then optimize Dt according to (2). Initialize distance list dt = [0, ..., 0],where length(dt) = t− 1. for j=1:t-1 do
Calculate dM (Dtl ,D j l ) in the last convolutional layer according to (8), and update dt[j] = dM (D t l ,D j l )
end for Sort dt in ascending order, and select least Ec indices to construct St. Finetune additional n heads on current task. Update Atom Memory MD ←− [MD,Dt].
end if end if
end for return MD, α.

heading: C EXPERIMENTAL DETAILS

heading: C.1 DATASET STATISTICS

text:
We provide the dataset statistics of 10-Split CIFAR100 used in class-incremental setting in Tab. A, and statistics of 20-Split CIFAR100and 20-Split miniImageNetin task-incremental setting in Tab. B and Tab. C.
Table A: Statistics of 10-Split CIFAR100.
10-Split CIFAR100 # tasks 10
Img. Size 32× 32× 3 # tasks/task 10
# Training samples/task 4500 # Validation samples/task 500
# Test samples/task 100

heading: C.2 NETWORK ARCHITECTURE

heading: C.2.1 ARCHITECTURE FOR CLASS-INCREMENTAL LEARNING

text:
We adopt the ResNet-32 as in (Rebuffi et al., 2017), which includes 31 convolutional layers and 1 fully connected layer. We substitute every convolutional layer with the one has the proposed decomposed filters (m = 12) that has the same number of input/ output channels.
Table B: Statistics of 20-Split CIFAR100and 20-Split miniImageNet.
20-Split CIFAR100 20-Split miniImageNet # tasks 20 20
Img. Size 32× 32× 3 84× 84× 3 # tasks/task 5 5
# Training samples/task 2,125 2,125 # Validation samples/task 375 375
# Test samples/task 500 500
Table C: Statistics of 5-Split MNIST.
Task (0,1) (2,3) (4,5) (6,7) (8,9) # Training samples/task 10,766 10,276 9,574 10,356 10,030
# Validation samples/task 1,899 1,813 1,689 1,827 1,770 # Test samples/task 2,115 2,042 1,874 1,986 1,983

heading: C.2.2 ARCHITECTURE FOR TASK-INCREMENTAL LEARNING

text:
Architecture for 5-Split MNIST. Conv(k=3, m=12, 16)-ReLU-Dropout(0.2)-MaxPool(2)-Conv(k=3, m=12,32)-ReLU-Dropout(0.2)-MaxPool(2)-Conv(k=3, m=12, 64)-ReLU-Dropout(0.2)-GAP-FC(32)-ReLUDropout(0.5)-FC(2),
where GAP stands for global average poling, Conv(k, m, c) means the convolution layer with kernel size k, m atoms, c output channels, Dropout(p) indicates the dropout layer with probability p, and FC(c) is the fully-connect layer with c output channels.
Architecture for 20-Split CIFAR100and 20-Split miniImageNet. Conv(k=3, m=12, 64)-ReLUMaxPool(2)-Conv(k=3, m=12, 192)-ReLU-MaxPool(2)-Conv(k=3, m=12, 384)-ReLU-Conv(k=3, m=12, 256)- ReLU-Conv(k=3, m=12, 256)-ReLU-MaxPool(2)-GAP-Dropout(0.5)-FC(5)

heading: C.3 TRAINING DETAILS

heading: C.3.1 CLASS-INCREMENTAL LEARNING

text: For CIFAR100, we choose SGD with batch-size of 128, learning rate of 0.01, momentum of 0.9 and weight decay 1e-3. The model is trained for 250 epochs, with learning rate drop by 0.1 at the 100-th and 200-th epoch. For ImageNet-Subset, we choose SGD with batch-size of 128, learning rate of 0.05, momentum of 0.9, and weight decay 1e-4. The model is trained for 150 epochs, with learning rate drop by 0.1 at the 90-th and 120-th epoch. For CUBS and Flowers, We choose SGD with momentum 0.9, batch-size of 64, and weight decay of 1e-4. On CUBS, the model is finetuned for 200 epochs with learning rate of 1e-2 (drop by 0.1 at the 100th and 160th epoch) on the first task, and trained for 120 epochs with learning rate of 5e-3 (drop by 0.1 at the 70th epoch) on task 2-6. On Flowers, the model is finetuned for 120 epochs with learning rate of 1e-2 (decay by 0.1 at the 50th and 90th epoch) on the first task, and trained for 70 epochs with learning rate of 1e-3 (decay by 0.1 at the 40th epoch) on task 2-6.

heading: C.3.2 TASK-INCREMENTAL LEARNING

text: We choose SGD with batch-size of 64 for all experiments under the task-incremental setting. For 5-Split MNIST, we train the model for 20 epochs with learning rate of 0.001. For 20-Split CIFAR100, we train the model for 100 epochs with learning rate of 0.001, which drops by 0.1 at the 60-th epoch. For 20-Split miniImageNet, we train the model for 200 epochs with learning rate of 0.001, which drops at the 100-th and 150-th epoch.

heading: C.4 METRIC

text:
ACC and BWT used in the task-incremental settings can be expressed as,
ACC = 1
T T∑ i=1 AccT,i, BWT = 1 T − 1 T−1∑ i=1 AccT,i − Acci,i, (2)
where T is total number of tasks, Acci,j is performance on j-th task after learning i-th task (Lopez-Paz & Ranzato, 2017).
Figure A: Correlation analysis between task similarities calculated with CCA (Raghu et al., 2017) and task similarities assessed with filter subspace distance.

heading: C.5 CORRELATION ANALYSIS BETWEEN CCA AND FILTER SUBSPACE DISTANCE.

text: We conduct the analysis between the task similarities measured with CCA (Raghu et al., 2017) on deep feature spaces, and the proposed task similarities assessed with Grassmann distance on filter subspaces. The analysis is conducted on 20-Split CIFAR100with the proposed model. We randomly select 50 task pairs from ( 20 2 ) task pairs. Given a task pair, we use 500 images from each task to calculate the CCA similarity, and use the inverse of their Grassmann distance between their last layers’ filter subspaces as the filter-subspace similarity. As shown in Fig. A, filter-subspace similarities quite correlates to CCA similarities which we deem as the golden standard, while they are significantly more efficient to calculate with our atom continual learning framework.

heading: C.6 ADDITIONAL TASK-INCREMENTAL EXPERIMENTAL RESULTS

text:
We provide additional results on the standard 5-Split MNIST dataset. Note that due to the simplicity of 5-Split MNIST, we only apply on it our base model, which is composed by cross-task coefficient sharing and atom swapping only. As shown in Tab. D, our base method outperforms both regularization-based methods and memory-based methods in terms of ACC. Besides, the proposed method demonstrates a guarantee against forgetting reflected by 0 in BWT. Note that the proposed base model is much smaller in size compared to the other methods. Moreover, comparing to GEM Lopez-Paz & Ranzato (2017) and VCL Nguyen et al. (2018), our base network adopts far smaller memory for storing merely filter atoms for 3 layers, each of them has a hundred of parameters only.
Table D: Results on 5-Split MNIST.
Method ACC% BWT% Arch. Size Memory(MB) (MB)
EWC (Kirkpatrick et al., 2017a) 95.78± 0.35 -4.20± 0.21 1.1 - HAT (Serra et al., 2018) 99.59± 0.01 0.00± 0.04 1.1 -
UCB (Ebrahimi et al., 2019) 99.63± 0.02 0.00± 0.00 2.2 - VCL (Nguyen et al., 2018) 95.97± 1.03 -4.62± 1.28 1.1 -
GEM (Lopez-Paz & Ranzato, 2017) 94.34± 0.82 -2.01± 0.05 6.5 0.63 VCL-C (Nguyen et al., 2018) 93.6± 0.2 -3.10± 0.20 1.7 0.63 ACL (Ebrahimi et al., 2020) 99.76± 0.03 0.01± 0.01 1.6 - MIR (Aljundi et al., 2019a) 87.60± 0.70 7.00± 0.90 5.9 0.71 GSS (Aljundi et al., 2019b) 84.80± 1.80 - 3.7 0.58
Ours (Base) 99.84± 0.05 0.00± 0.00 0.24 0.04

heading: C.7 ADDITIONAL CLASS-INCREMENTAL RESULTS

text:
C.7.1 ADDITIONAL RESULTS ON 10-Split CIFAR100
We provide results 10-Split CIFAR100 under class-incremental setting with the ResNet-18 (He et al., 2016) in Tab. E and Fig. B. Our method achieves the best result with a large margin.
Table E: Class-incremental results on 10-Split CIFAR100.
Method Acc. (avg.) Acc. (last)
LwF (Li & Hoiem, 2017) 44.5 23.9 EWC (Kirkpatrick et al., 2017a) 36.2 16.4 SI (Zenke et al., 2017a) 37.8 23.3 MAS (Aljundi et al., 2018b) 33.4 15.4 RWalk (Chaudhry et al., 2018b) 35.2 17.9 DMC (Zhang et al., 2020) 57.1 36.2
Ours (Base) 58.6 40.57 w/ E = 2 62.4 46.37 w/ E = 3 62.9 46.50
Figure B: Class incremental learning results on 10-Split CIFAR100.
C.7.2 ADDITIONAL RESULTS ON 10-Split ImageNet-Subset
We further validate our method on a large-scale image classification dataset, ImageNet-Subset, which is a subset of the ImageNet dataset that contains the first 100 classes. As in Masanaet et al. (2020), we split 100 classes into 10 tasks with 10 classes per task. We adopt the ResNet-18 as the architecture and substitute the convolutional layers with our decomposed version. Training details are provided in Appendix C.3.
Table F: Class-incremental results on 10-Split ImageNet-Subset.
Method Memory Acc. (last)
iCaRL (Rebuffi et al., 2017)
301.4 MB 43.6 EEIL (Castro et al., 2018) 36.6 BiC (Wu et al., 2019) 45.6 IL2M (Belouadah & Popescu, 2019) 38.2
Ours (Base) 7.3 MB 43.9 w/ Ew = 2 7.7 MB 46.3
We benchmark our method with existing methods that store 2000 exemplars in the external memory. Their results are faithfully reproduced in Masanaet et al. (2020). As shown in the Tab. F, our method with intra-task ensemble achieves the best result with significantly less memory usage.

heading: C.7.3 ADDITIONAL RESULTS ON CUBS AND FLOWERS

text:
We further validate our methods on CUBS (Welinder et al., 2010) and Flowers (Nilsback & Zisserman, 2008) datasets. Following Yu et al. (2020), we use the random seed (1993) to select 100/50 classes as the first task, and
evenly split the rest 100/50 classes into 5 tasks. We adopt ResNet-18 pretrained on ImageNet with substituted decomposed convolutional layers. Training details are illustrated in Appendix C.3.
Table G: Average incremental accuracy on CUBS and Flowers datsets
.
Method CUBS Flowers
LwF-E∗ (Li & Hoiem, 2017) 69.8 87.2 EWC-E∗ (Kirkpatrick et al., 2017a) 69.7 85.9 MAS-E∗ (Aljundi et al., 2018b) 68.5 84.7 SDC (Yu et al., 2020) 70.0 86.8
Ours (Base) 69.3 86.9 w/ Ew = 2 72.1 89.3
Under the class-incremental setting, we compare the average incremental accuracy with other benchmarks. On both datasets, as shown in the table above, our method produces the best results.

heading: C.8 ABLATION STUDY ON DISTANCE METRICS IN INTER-TASK ENSEMBLE

text:
We present an ablation study of different distance metrics including the Grassmann distance, L2 norm, and cosine distance for inter-task ensemble. As shown in Tab. H, inter-task ensemble with Grassmann distance achieves the best result.
Table H: Ablation study on different metrics in inter-task ensemble.
Method CIFAR-100 miniImageNet
Ours (Base) 79.13 66.01
w/ Ec = 3 (L2 norm) 80.35 67.13 w/ Ec = 3 (cosine) 80.11 66.82 w/ Ec = 3 (Grassmann) 80.75 67.84
C.9 INFERENCE TIME COMPARISONS
On the CIFAR-100 dataset, we compare inference time on all classes with other methods. All methods are tested on a single RTX 2080ti GPU under the class-incremental setting. Our method has comparable inference time to other methods, as shown in the Tab. I.

heading: C.10 ABLATION STUDY OF SENSITIVITY OF FIRST TASK.

text: The continual learning literature mostly sample tasks from a single dataset [4,5,6]. In this commonly adopted scenario, we observe that our model is not sensitive to the selection of the first task. we provide an ablation study on 20-Split CIFAR-100. As suggested, we perform experiments with 20 different selections of the first task and calculate the statistics of results of 20 runs. Our base method achieves ACC = 79.41± 0.92, which shows a small variance.

heading: C.11 MEMORY CALCULATION

heading: C.11.1 CLASS-INCREMENTAL LEARNING

text:
We calculate the external memory usage for exemplar-based methods with 2000 exemplars, e.g., Rebuffi et al. (2017), and our method with atom memory.
• 2000 exemplars: 2000 ∗ (32 ∗ 32 ∗ 3 + 1)/1e6 = 6.15 MB. • Our (Base): (31 ∗ 1 ∗ 12 ∗ 9 + 64 ∗ 10) ∗ 4/1e6 ∗ 9 = 0.14 MB. • Our (w/ Ew = 2): (31 ∗ 2 ∗ 12 ∗ 9 + 64 ∗ 10 ∗ 2) ∗ 4/1e6 ∗ 9 = 0.29 MB.

heading: C.11.2 TASK-INCREMENTAL LEARNING

text:
We calculate external memory usage for our method in 20-Split CIFAR100 and 20-Split miniImageNet below.
• Our (Base): (5 ∗ 12 ∗ 9 + 256 ∗ 5)/1e6 ∗ 4 ∗ 19 = 0.14 MB • Our (w/ Ec = 3): (5 ∗ 12 ∗ 9 + 256 ∗ 5 + 256 ∗ 5 ∗ 3)/1e6 ∗ 4 ∗ 19 = 0.43 MB • Our (w/ Ew = 2): (5 ∗ 12 ∗ 9 ∗ 2 + 256 ∗ 5 ∗ 2)/1e6 ∗ 4 ∗ 19 = 0.28 MB • Our (w/ both): (5 ∗ 12 ∗ 9 ∗ 2 + 256 ∗ 5 ∗ 2 + 256 ∗ 5 ∗ 2 ∗ 3)/1e6 ∗ 4 ∗ 19 = 0.86 MB