summaries: This paper proposes a method for continual learning. The model has three components: a) a data generator to be used at training time to replay past examples, b) a parameter generator that takes the input observation to produce parameters for c) the actual classifier. The authors demonstrate the method on simple datasets with a stream of 2 or 3 tasks. This paper proposes a Dynamic Parameter Generator (DPG) that given a test input modifies the parameters of a classification model. They also propose to regularize the training using a Data Generator (DG) to slow down catastrophic forgetting. DG is used to constrain the training that the internal representations of data generated by DG does not rapidly change. DG removes the need for storage of data or labels.
abstractText: Learning multiple tasks sequentially is important for the development of AI and lifelong learning systems. However, standard neural network architectures suffer from catastrophic forgetting which makes it difficult for them to learn a sequence of tasks. Several continual learning methods have been proposed to address the problem. In this paper, we propose a very different approach, called Parameter Generation and Model Adaptation (PGMA), to dealing with the problem. The proposed approach learns to build a model, called the solver, with two sets of parameters. The first set is shared by all tasks learned so far and the second set is dynamically generated to adapt the solver to suit each test example in order to classify it. Extensive experiments have been carried out to demonstrate the effectiveness of the proposed approach.
authors.name: Wenpeng Hu
authors.name: Zhou Lin
authors.name: Bing Liu
authors.name: Chongyang Tao
authors.name: Zhengwei Tao
authors.name: Dongyan Zhao
authors.name: Jinwen Ma
authors.name: Rui Yan
references.authors: Rahaf Aljundi Francesca Babiloni Mohamed Elhoseiny Marcus Rohrbach Tinne Tuytelaars
references.title: Memory aware synapses: Learning what (not) to forget
references.venue: arXiv preprint arXiv:1711.09601,
references.authors: Andrew Brock Theodore Lim James M Ritchie Nick Weston
references.title: Smash: one-shot model architecture search through hypernetworks
references.venue: arXiv preprint arXiv:1708.05344,
references.authors: Raffaello Camoriano Giulia Pasquale Carlo Ciliberto Lorenzo Natale Lorenzo Rosasco Giorgio Metta
references.title: Incremental robot learning of new objects with fixed update time
references.venue: In Robotics and Automation (ICRA),
references.authors: Zhiyuan Chen Bing Liu
references.title: Lifelong machine learning
references.authors: Misha Denil Babak Shakibi Laurent Dinh Nando De Freitas
references.title: Predicting parameters in deep learning
references.venue: In Advances in neural information processing systems,
references.authors: Chrisantha Fernando Dylan Banarse Charles Blundell Yori Zwols David Ha Andrei A Rusu Alexander Pritzel Daan Wierstra
references.title: Pathnet: Evolution channels gradient descent in super neural networks
references.venue: arXiv preprint arXiv:1701.08734,
references.authors: Tommaso Furlanello Jiaping Zhao Andrew M Saxe Laurent Itti Bosco S Tjan
references.title: Active long term memory networks
references.venue: arXiv preprint arXiv:1606.02355,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Xu He Herbert Jaeger
references.title: Overcoming catastrophic interference using conceptor-aided backpropagation
references.authors: Geoffrey Hinton Oriol Vinyals Jeff Dean
references.title: Distilling the knowledge in a neural network
references.venue: arXiv preprint arXiv:1503.02531,
references.authors: Gao Huang Zhuang Liu Laurens Van Der Maaten Kilian Q Weinberger
references.title: Densely connected convolutional networks
references.venue: In CVPR,
references.authors: Diederik P Kingma Max Welling
references.title: Auto-encoding variational bayes
references.venue: arXiv preprint arXiv:1312.6114,
references.authors: James Kirkpatrick Razvan Pascanu Neil Rabinowitz Joel Veness Guillaume Desjardins Andrei A Rusu Kieran Milan John Quan Tiago Ramalho Agnieszka Grabska-Barwinska
references.title: Overcoming catastrophic forgetting in neural networks
references.venue: Proceedings of the national academy of sciences,
references.authors: Sang-Woo Lee Jin-Hwa Kim Jaehyun Jun Jung-Woo Ha Byoung-Tak Zhang
references.title: Overcoming catastrophic forgetting by incremental moment matching
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Jens Lehmann Robert Isele Max Jakob Anja Jentzsch Dimitris Kontokostas Pablo N Mendes Sebastian Hellmann Mohamed Morsey Patrick Van Kleef Sören Auer
references.title: Dbpedia–a largescale, multilingual knowledge base extracted from wikipedia
references.venue: Semantic Web,
references.authors: Jingyang Li Maosong Sun Xian Zhang
references.title: A comparison and semi-quantitative analysis of words and character-bigrams as features in chinese text categorization
references.venue: In ACL,
references.authors: Shen Li Zhe Zhao Renfen Hu Wensi Li Tao Liu Xiaoyong Du
references.title: Analogical reasoning on chinese morphological and semantic relations
references.venue: In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics
references.authors: David Lopez-Paz
references.title: Gradient episodic memory for continual learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Nicolas Y Masse Gregory D Grant David J Freedman
references.title: Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
references.venue: arXiv preprint arXiv:1802.01569,
references.authors: Michael McCloskey Neal J Cohen
references.title: Catastrophic interference in connectionist networks: The sequential learning problem
references.venue: In Psychology of learning and motivation,
references.authors: German I Parisi Ronald Kemker Jose L Part Christopher Kanan Stefan Wermter
references.title: Continual lifelong learning with neural networks: A review
references.venue: arXiv preprint arXiv:1802.07569,
references.authors: Jeffrey Pennington Richard Socher Christopher D. Manning
references.title: Glove: Global vectors for word representation
references.venue: In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543,
references.authors: Sylvestre-Alvise Rebuffi Alexander Kolesnikov Georg Sperl Christoph H Lampert
references.title: icarl: Incremental classifier and representation learning
references.venue: In Proc. CVPR,
references.authors: Jonathan Schwarz Jelena Luketina Wojciech M Czarnecki Agnieszka Grabska-Barwinska Yee Whye Teh Razvan Pascanu Raia Hadsell
references.title: Progress & compress: A scalable framework for continual learning
references.venue: arXiv preprint arXiv:1805.06370,
references.authors: Ari Seff Alex Beatson Daniel Suo Han Liu
references.title: Continual learning in generative adversarial nets
references.venue: arXiv preprint arXiv:1705.08395,
references.authors: Joan Serrà Dı́dac Surı́s Marius Miron Alexandros Karatzoglou
references.title: Overcoming catastrophic forgetting with hard attention to the task
references.venue: arXiv preprint arXiv:1801.01423,
references.authors: Hanul Shin Jung Kwon Lee Jaehong Kim Jiwon Kim
references.title: Continual learning with deep generative replay
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Ilya Sutskever Oriol Vinyals Quoc V Le
references.title: Sequence to sequence learning with neural networks. In Advances in neural information processing
references.authors: Ragav Venkatesan Hemanth Venkateswara Sethuraman Panchanathan Baoxin Li
references.title: A strategy for an uncompromising incremental learner
references.venue: arXiv preprint arXiv:1705.00744,
references.authors: Wenhui Wang Nan Yang Furu Wei Baobao Chang Ming Zhou
references.title: Gated self-matching networks for reading comprehension and question answering
references.venue: In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
references.authors: Jaehong Yoon Eunho Yang Jeongtae Lee Sung Ju Hwang
references.title: Lifelong learning with dynamically expandable
references.authors: Friedemann Zenke Ben Poole Surya Ganguli
references.title: Continual learning through synaptic intelligence
references.venue: arXiv preprint arXiv:1703.04200,
sections.heading: 1 INTRODUCTION
sections.text: It is well-known that neural networks (NNs) suffer from catastrophic forgetting (CF) (McCloskey & Cohen, 1989), which refers to the phenomenon that when learning a sequence of tasks, the learning of each new task may cause the NN to forget the models learned for the previous tasks. Without solving this problem, an NN is hard to adapt to lifelong or continual learning, which is important for AI.
  Problem Statement: Given a sequence of supervised learning tasks T = (T1, T2, . . . , TN ), we want to learn them one by one in the given sequence such that the learning of each new task will not forget the models learned for the previous tasks.
  In recent years, many approaches (often called continual learning) have been proposed to lessen the effect of CF (Chen & Liu, 2018; Parisi et al., 2018), e.g., dynamically expandable network (DEN) (Yoon et al., 2018), learning without forgetting (LWF) (Li & Hoiem, 2016)), elastic weight consolidation (EWC) (Kirkpatrick et al., 2017), incremental moment matching (IMM) (Lee et al., 2017), gradient episodic memory (GEM) (Lopez-Paz et al., 2017), generative replay (GR) (Shin et al., 2017), etc. These existing studies except DEN and LWF focused on learning a model parameterized by a single joint set of parameters θ∗ which is assumed to work well for all tasks. We call them joint parameterization (JP) methods. DEN and LWF require constantly increasing the number of parameters and thus can result in a huge and complex model.
  JP methods, however, suffer from accuracy deterioration. Assume we have two tasks A and B that need to be learned sequentially. Let θ∗A and θ ∗ B be the optimal parameters for Task A and Task B respectively when each of them is learned individually. Let θ∗ be the joint parameters learned by the existing approaches to perform Tasks A and B in sequence. Inevitably, θ∗ is different from θ∗A and/or θ∗B and is highly likely to result in more errors for the two tasks than θ ∗ A and θ ∗ B individually.
  ∗Equal contribution †Corresponding author
  In this paper, we propose a different approach, called Parameter Generation and Model Adaptation (PGMA), to dealing with CF and to significantly reducing accuracy deterioration. PGMA does not learn a joint parameter set θ∗. Instead, it learns a parameter generator f(·) and a shared parameter set θ0. The key idea of PGMA is as follows: The overall classification network, called the solver S, has two disjoint subsets/parts of parameters. The first subset is θ0, which is shared by all tasks learned so far. The second subset is just a place holder H that will be filled by parameters generated by f(·) for each test instance. That is, given a test instance, a set of parameters will be generated by the learned parameter generator f(·) to replace H . Solver S then combines θ0 and the generated parameters for the test instance to classify it. Clearly, unlike existing JP approaches, we do not have a network with a set of fixed parameters for S. The idea is that θ0 contains the common features of all tasks, and the generated parameters for each test instance adapt S for each test instance in order to classify it.
  Since f(·) and the shared parameters θ0 will change during training for each new task, forgetting can occur for previous tasks. To deal with it, in training f(·) and S for each task Ti, in addition to the training data of Ti, a small number of replayed samples will be generated by a data generation network for the previous tasks to ensure that the knowledge learned for previous tasks remain stable/unforgotten. Compared with the existing generative replay methods, our method does not need labels for the replayed data because we use the replayed data only to constrain the training of S and f(·) rather than to treat them as surrogates of labeled training data from previous tasks. Further, since in the existing replay methods, the labels are produced by the current learned network itself, the labels can be noisy and biased, which may result in errors being accumulated and propagated to subsequent tasks. Our method does not have this problem.
  Apart from reducing the effect of accuracy deterioration of the existing approaches, the proposed approach also has some other advantages. First, no parameter increase or network expansion is needed to learn new tasks. Second, no previous data needs to be stored to enable the system to remember the previously learned models or knowledge.
  Experiments conducted using two image datasets (MNIST and CIFAR-10) and two text datasets (DBPedia ontology (Lehmann et al., 2015) and THUCNews (Li et al., 2006)) show that the proposed approach PGMA works well for different scenarios and different types of datasets, and outperforms the existing strong baselines markedly.
sections.heading: 2 PROPOSED PGMA FRAMEWORK
sections.text: Let the sequence of supervised learning tasks be T = (T1, T2, . . . , TN ). Each task Ti is represented by Ti = {xij , yij |j ∈ (1, . . . , Ni)}, where xij is the j-th example/sample of Ti, and yij is its label. We use (xt, yt) to denote a test instance/example. To simplify the notation, we will just use (xi, yi) to denote a training example from task Ti, omitting the second subscript j. Note that in the paper, we use the terms example, sample, and instance interchangeably.
  The proposed parameter generation and model adaptation (PGMA) architecture has three main components:
  • Solver S: It is the main classification model. As mentioned in Section 1, the parameter set of S consists of two subsets, θ0 that is shared by all tasks (and instances) and H , a parameter place holder, which will be replaced by the generated parameters set pi (or pt) for each training (or testing) example xi (or xt). pi (or pt) basically serves to adapt the solver S to classify the example in training or testing. This is the key idea of our approach. We adopt this parameter split as several studies (Yoon et al., 2018; Li & Hoiem, 2016; Kirkpatrick et al., 2017) have shown that only part of parameters of a neural network needs to be adjusted when learning a new task. See Section 2.2 • Dynamic Parameter Generator (DPG) f(·): It takes the embedding zi (or zt) of each input training (or testing) example xi (or xt) to generate the parameters pi (or pt) for solver S. Note that we use zi (or zt) rather than the raw data xi (or xt) because the raw data’s dimension can be very high. The embedding zi (or zt) as a low dimensional dense representation reduces the mapping space for DPG and thus reduces the difficulty in its parameter generation. See Section 2.2. • Data Generator (DG): It has two functions. The main function is to generate a set of replayed data or samples {x′m}Mm=1 using its decoder DGD for previous tasks to deal with catastrophic forgetting. The other function is to generate the embedding zi (or zt) of each input training (or testing) example xi (or xt) using its encoder DGE . See Section 2.3.
sections.heading: 2.1 OVERALL APPROACH OF PGMA
sections.text: We work backward by describing testing first before training. Given a test instance xt, DGE first generates its embedding zt, which is fed to f(·) to generate a set of parameters pt. Solver S then takes xt as input and uses the trained/learned shared parameters θ0 and pt to classify xt. θ0 contains the common features of all tasks learned so far. pt simply adapts S for xt in order to classify xt.
  For training, the pipeline of the proposed PGMA framework is shown in Figure 1. Given a new task Ti with its data (xi, yi), solver S and DPG f(·) are jointly trained to learn Ti and also not to forget the previously learned tasks. In each iteration, a set of parameters pi is generated by the current DPG f(zi, µ) for each training instance xi (zi being its embedding), where µ is the set of parameters of the DPG network, which is trained.
  In training the solver S and DPG for the new task Ti, both f(·) and the shared parameters θ0 will change, which can cause forgetting in DPG for previous tasks. To keep DPG remembering the acquired knowledge for previous tasks, we minimize the variation of certain layers’ output caused by the changes of θ0 and f(·) using the set of replayed samples {x′m}Mm=1 generated by DG.
  PGMA has two training components: (1) DPG and solver S (DPG&S for short), i.e., DPG is optimized together with solver S. (2) DG. The two components have their respective objective functions (see Sections 2.2 and 2.3), and are trained alternately. This is because DPG takes input zi (which is produced by DG) to generate parameters, and alternating training ensures consistent convergence rates of DG and DPG.
sections.heading: 2.2 DYNAMIC PARAMETER GENERATOR (DPG) AND SOLVER S IMPLEMENTATION
sections.text: Several neural networks can be used to implement DPG for parameter generation, e.g., convolutional neural network (CNN), recurrent neural network (RNN) and multilayer perceptron (MLP). The objective of this paper is not to explore all possible implementations. We found that a straightforward implementation using MLP can already achieve good results. Formally, DPG can be written as:
  pi = f(zi, µ) = σ(wDzi + b) (1)
  where σ is the activation function, and wD and b are the parameters of DPG and denoted by µ.
  For the implementation of solver S, several deep learning networks can be used as well. Again, a MLP is employed in this work. Each layer of the MLP is a perceptron and can be formalized by output = σ(wxinput), where xinput denotes the input of the particular perceptron. We also call a perceptron a basic unit 1. In general, for each basic unit k of the solver S, we can have a shared portion of the parameters θ0,k and a generated portion of the parameters pi,k, i.e.,
  θ∗i = combine(θ0,pi) = {[w∗i,k]}Kk=1 = {[θ0,k;pi,k]}Kk=1 (2)
  where combine(.) is concatenation and K is the number of basic units that the solver has. In our case, K is the number of hidden layers (basic units) of the solver MLP.
  It is important to note that the above is the most general case. In practice, there is no need to adapt all the basic units in the solver, but only a subset of them. In our experiments (Section 3), we will see that adapting only the final layer of the solver MLP can already achieve good results.
  1This is because most of the existing networks can be regarded as the composition of multiple basic units, even for those with complex structures, e.g., seq2seq (Sutskever et al., 2014), R-Net (Wang et al., 2017), Resnet (He et al., 2016) and Densenet (Huang et al., 2017). Further, complex structures give us more freedom to decide which part of the parameters should be generated.
  However, one may still ask whether the proposed combination method has sufficient capacity to adjust the solver S in general because only part of the parameters are generated. Actually, pik can affect all dimensions of the output of its corresponding basic unit k:
  w∗i,kxinput = [θ0,k;pi,k]xinput = θ0,kx 1 input + pi,kx 2 input (3)
  where x1input and x 2 input are the block vectors of xinput. pi,kx 2 input can be regarded as the bias and can adapt the output vector to any point in the vector space.
  To train DPG&S, we use cross entropy loss (Lce). The objective function of the solver S including DPG f(zi, µ) and θ0 for learning each new classification task Ti is defined as:
  minimize µ,θ0
  Lce(S(xi, θ∗i ), yi)
  s.t. M∑ m=1 ||R(x′m, θ∗i )−R(x′m, θ∗i−1)|| < r (4)
  where Lce is the cross entropy loss,R(·) denotes the output of all basic units (which can be any layer or a set of layers2) in solver S, which we will introduce later, and θ∗i is the whole set of parameters of S (θ∗i = combine(pi, θ0)) representing the adaptation of S. We can see that the generated replayed samples x′m are used as constraints to alleviate DPG&S’s forgetting. Specially, we extend the knowledge distillation loss (Hinton et al., 2015) to the general situation. That is, to keep the past learned knowledge, the output of the basic units in the solver should not change much when learning a new task with the help of the generated data. If we do not consider the activation function, the constraints in Eq. 4 can also be written as:
  min M∑ m=1 K∑ k=1 ||w∗i,kx′m,k −w∗i−1,kx′m,k|| (5)
  where K again denotes the number of basic units and M denotes the number of replayed samples. The basic unit with a smaller k is at the relative lower layer of the solver network.3 x′m,k is the input of the kth basic unit and is calculated through forward propagation, except x′m,1 = DGD(z sample m , θ ′ d) which is the initial replayed sample x′m generated by DG (before optimizing the current task Ti). w∗i,k is the combined parameter introduced in Section 2.2.
  Two questions may be asked about Eqs. 4 and 5. (1) Since the replayed data x′ (denoting all x′m) is used in Eqs. 4 and 5, does it play the same role as the original data x from previous tasks? (2) What is the impact of the constraints on learning the new task? We answer the two questions now.
  Question 1: The answer is positive. Since the purpose of the constraints is to maintain the learned effect of the old data, clearly, using the original old data x in the constraints is better. However, we proved the positive answer to question 1, which is given in Appendix.
  Question 2: In calculating Eq. 5, if we stack all {x′m,k}Mm=1 (column vectors) into a matrix X′k, Eq. 5 can be regarded as constraining w∗i,kX ′ k to remain unchanged. If X ′ k is a row low rank matrix, w∗i,k can be trained to fit the new tasks. Otherwise, especially if X ′ k is a row full rank matrix, w ∗ i,k cannot be trained and therefore cannot learn new tasks. However, benefiting from dynamic parameter generation, our approach will not suffer from this problem. That is because w∗i,k is specially generated through DPG to perform well with input X′k. And the new parameters will be generated to fit new tasks while w∗i,k is able to remain unchanged.
sections.heading: 2.3 DATA GENERATOR (DG)
sections.text: As indicated earlier, DG has two functions. First, it compresses the original input data xi to zi using its encoder to reduce the number of dimensions of xi and consequently reduces the mapping space of DPG to make the generation of the parameters pi easier. The compression is formulated by:
  zi = DGE(xi, θe) (6) 2Note that in this case, the parameters in R is a subset of parameters (θ∗i ) for solver S. To simplify the
  expression and without causing confusion, we don’t introduce new symbols and let the symbols unchanged. 3Note that constraining only the units in the last layer can already achieve good results.
  where DGE is the encoder of DG with parameters θe.
  Second, it generates the replayed data for previous tasks to deal with forgetting in solver S and DPG. Note that DG differs from data generators in the existing generative replay (GR) methods (Shin et al., 2017) as GR needs to generate both the replay data x′m and their labels y ′ m (using the solver learned so far), which can be noisy. DG only generates the data x′m but not the labels (which are not needed by our approach), and then the labeling errors won’t affect our model PGMA, but will hurt GR.
  Each replayed sample x′m is generated by the decoder of DG, called DGD:
  x′m = DGD(z sample m , θ ′ d) (7)
  where zsamplem is the m th sample sampled from the multivariate normal distribution.4 θ′d is the set of parameters of DGD before optimizing the current task Ti.
  DG can be implemented with an auto-encoder, e.g., VAE-like (Variational Auto-Encoder (Kingma & Welling, 2013)) and WAE-like (Wasserstein Auto-Encoder) auto-encoders. We use WAE in DG since it can let different examples get a chance to stay far away from each other, which promotes better reconstruction (Tolstikhin et al., 2017).5 Note that DG also suffers from forgetting, which is dealt with like DPG and also using specially designed losses (see below).
  To train DG, we use mean square error as the reconstruction loss to enable its replay ability of the past data, and add to it the penalized form of the Wasserstein distance between the distribution of zi and multivariate normal distribution to help generate data (Tolstikhin et al., 2017) (together denoted as Lwae).
  Note also since we only have one data generator DG, it has the forgetting problem caused by incremental training of new tasks too. To avoid forgetting in DG, we investigated using loss functions to constrain DG to overcome its forgetting as shown in Figure 2. We describe this approach here:
  min M∑ m=1 ||zsamplem −DGE(x′m, θe)|| (8)
  min M∑ m=1 ||DGD(zsamplem , θd)− x′m|| (9)
  where x′m is the replayed data (see Eq. 7), θe and θd are the encoder’s and decoder’s parameters of DG in the process of learning the new task Ti, respectively, and zsamplem and M have the same meanings as they are in DPG&S. Eq. 8 constrains the consistency of DG’s decoder and encoder over the randomly sampled zsamplem . Eq. 9 ensures that DG’s decoder can still remember the old data. Using Eqs. 8 and 9, we can maintain DG’s ability to reflect the old data. Overall, the final objective function for DG is composed by Lwae, Eq. 8 and 9.
sections.heading: 2.4 THE COMPLETE TRAINING PROCEDURE
sections.text: Finally, we summarize the whole training procedure of PGMA in Algorithm 1, which is selfexplanatory. “//” is followed by comments. The test procedure is straightforward, see Section 2.1.
  4In training, we force zi to satisfy the multivariate normal distribution. We can then sample many zsamplem from the multivariate normal distribution.
  5Our approach is not limited to using WAE. Other auto-encoders may also be applied.
  Algorithm 1 PGMA (Parameter Generation and Model Adaptation) training
  Input: T = {Ti}Ni=1, where Ti = {xij , yij} Initial: Randomly initialize f(·), DG, S; // Learning the first task for all n = 0, . . . , until convergence do
  5: Sample a mini-batch from T1; // Training DG Minimize Lwae and then update DG; // Training DPG and S. pn and Sn below // denote a batch of generated parameters
  10: // and adapted solvers, respectively Compute pn using Eqs. 1 & 6; // Section 2.2 Construct Sn using pn and θ0; // Section 2.2 Minimize Lce then update f(·) and S;
  end for
  15: // Learning the subsequent tasks for all i = 2; i ≤ N ; i++ do
  Generate replayed samples x′m by DG; for all n = 0, . . . , until convergence do
  Sample a mini-batch from Ti; 20: Minimize Lwae, Eqs. 8 &9 and then update
  DG; // Section 2.3 Compute pn using Eqs. 1 & 6; Construct Sn using pn and θ0; Minimize Lce, Eq. 5, and then update f(·) and S; // Section 2.2
  end for 25: end for
sections.heading: 3 EXPERIMENTS
sections.text: We now evaluate the proposed approach PGMA6 and compare it with state-of-the-art baselines using two image datasets and two text datasets.
sections.heading: Datasets
sections.text: • Two image datasets: (1) MNIST: this dataset consists of 70,000 images of handwritten digits from 0 to 9. We use 60,000/3000/7000 images for training/validation/testing respectively. (2) CIFAR-10: this dataset consists of 60,000 32x32 color images of 10 classes, with 6000 images per class. There are 50,000/3000/7000 images for training/validation/testing respectively. • Two text datasets: (1) DBPedia ontology: this is a crowd-sourced dataset (Lehmann et al., 2015) with 560,000 training samples and 70,000 test samples. Out of the 70,000 test samples, we use 10,000 for validation and 60,000 for test. (2) THUCNews: this dataset consists of 65,000 sentences of 10 classes (Li et al., 2006). We randomly select 50,000/5000/10,000 sentences for training/validation/testing respectively.
sections.heading: Experiment Settings
sections.text: Data Preparation: To simulate sequential learning, we adopt the same two data processing methods as in (Lee et al., 2017), named disjoint and shuffled. • Disjoint: This method divides each dataset into several subsets of classes. Each subset is a task. For example, we divide the MNIST dataset into two tasks (or subsets of classes). The first task consists of digits (classes) {0, 1, 2, 3, 4} and the second task consists of the remaining digits (classes) {5, 6, 7, 8, 9}. The systems learn the two subsets as two tasks in a sequential fashion and regard them together as 10-class classification. In order to consider more tasks in testing, for MNIST, CIFAR-10, and THUCNews, which all have 10 classes, we created two experiment settings of 2 tasks (5 classes per task) and 5 tasks (2 classes per task). For DBPedia, which has 14 classes, we created three experiment settings, 2 tasks (7 classes per task), 3 tasks (5, 5, and 4 classes for the three tasks respectively), and 5 tasks (3, 3, 3, 3, and 2 classes for the 5 tasks respectively). • Shuffled: This method shuffles the input pixels of an image with a fixed random permutation. Two experiment settings were created: 3 tasks and 5 tasks. In both cases, the dataset for the first task is the original dataset. The datasets for the rest of the tasks are constructed through shuffling. Since shuffling of words in a sentence will change the sentence meaning and results in confusion, thus this experiment is not done on text datasets.
  Baselines: We use three state-of-the-art baselines that are representative of the current approaches: 1) EWC (elastic weight consolidation) (Kirkpatrick et al., 2017); 2) IMM (incremental moment matching) (Lee et al., 2017); 3) GR (generative replay) (Shin et al., 2017). We use the open source code released by the authors or the third party for comparison. We use Adam algorithm to update the parameters and also use Adam as a baseline to show how serious the forgetting problem is.
  6https://github.com/morning-dews/PGMA tensorflow
  Training Details: For fair comparison, our proposed approach uses the same solver (or classifier) as the baselines. That is, a multilayer perceptron is adopted as the solver/classifier (as the baselines all use this method), which is a 3-layer network (i.e., two basic units with each hidden layer as a unit) followed by a softmax layer. For our approach, the total number of parameters in the solver includes both the generated parameters p and the shared parameters θ0. Due to the differences among different datasets, we adopt different settings for them, see Table 5 in Appendix for details. All baselines and our approach use the same setting for the same dataset. We use a 3-layer perceptron (with 2 hidden layers) network (we also call it T-net) for DPG and set the size of each hidden layer to 1000. Each T-net can generate 100 parameters at a time. We can parallel several T-nets in the DPG to generate more parameters when needed. The network parameters are updated using the Adam algorithm with a learning rate of 0.001.
sections.heading: Results and Analysis
sections.text: Figure 3 shows the test accuracy plots per 40 training steps of each method for each task as the tasks are sequentially learned. IMM is not shown here because IMM works by combining well trained models of individual tasks to form one joint model for all tasks. Thus, we cannot draw an accuracy curve with increased training steps like others. However, its final results and those of the other systems are given in Tables 1 and 2. Shuffled CIFAR-10 is also not included in the figure because our experiments show that it cannot be learned by CNN. It is also not used in any existing paper. Note that due to space limitations, Figure 3 only plots the results of those settings with only 2 and 3 tasks.
  From Figure 3, we can make the following observations:
  (1). The proposed PGMA method consistently outperforms the baselines in overcoming forgetting, by a big margin in most cases.
  (2). EWC does not perform well for the disjoint setting. GR is better but still poorer than our method. Adam’s results show that forgetting is very serious for all datasets and settings.
  Time efficiency: Due to DPG, our method uses slightly more time than baselines, under 25% for training and under 28% for testing compared to GR, which is efficient. This is a small price to pay for the major gain in accuracy in dealing with catastrophic forgetting.
  Memory usage: There is no need for our method to save data, which results in a large memory saving. For more details about time efficiency and memory usage, please see Appendix (Section C).
  From Table 1, we can see that our PGMA method is consistently superior to EWC, GR, and IMM on different datasets with 2 or 3 tasks (5 tasks results are given in Table 2). We believe that is because our PGMA model can reduce the effect of the accuracy deterioration problem discussed in Section 1. Adam’s results show that forgetting is very serious as it does not deal with the forgetting problem. EWC’s performance is poor for the disjoint case (a more realistic setting in practice), which is also reported by other researchers (Lee et al., 2017). Appendix has the results for only the last task.
  Table 2 shows the results for 5 tasks. EWC and Adam are not included as they performed poorly. Again, we observe that our PGMA method is markedly better than the baselines.
sections.heading: Ablation Study
sections.text: Here we study how the system behaves with less and less parameters in θ0 or more and more parameters replaced by the parameters generated by DPG. We selected the disjoint MNIST 2 tasks setting to conduct the experiment as it is more useful than the shuffled setting and also more difficult.
  Table 3 shows the accuracy results (averaged in the same way as in Table 1) when a portion of the parameters in only the last layer of the solver is replaced by the parameters generated by DPG. We observe that the accuracy improves with increased percentages of parameters being replaced.
  The best accuracy is obtained when 80% of the parameters in the last layer are replaced through DPG, and the accuracy won’t further improve with more replaced parameters. This observation indicates that replacing a part of parameters in solver to adapt new input tasks is sufficient. The same conclusion can also be made by replacing the parameters in the first hidden layer of the solver (which has 2 hidden layers). We fix the replacing percentage of the last layer to 20%, and then increase the replacing percentages of the first layer. The best accuracy reaches 94.31%(±0.84%) when replacing 40% parameters of the first layer, which gains only 1.87% in accuracy compared with no replacement. This result indicates that it suffices to replace the parameters in the last layer.
  Table 4 shows the contribution of different components. We can see a significant drop if DPG is removed. The second row gives the performance of our model when only DG and constraints are used. The third row shows the result without using constraints (DG+label(like GR)) but replacing them with the replay method. We can see that constraints work better than predicting labels.
sections.heading: 4 RELATED WORK
sections.text: Many approaches have been proposed to deal with catastrophic forgetting (CF), which is one of the challenging problems of neural networks for lifelong learning (Chen & Liu, 2018). EWC (Kirkpatrick et al., 2017) quantifies the importance of weights to previous tasks, and selectively alters the learning rates of weights. Following EWC, Zenke et al. (2017) measured the synapse consolidation strength in an online fashion and used it as regularization. Learning without forgetting (LWF) (Li & Hoiem, 2016) feeds the old network with new training data in new tasks and regards the output as ”pseudolabels”. In Incremental Moment Matching (IMM) (Lee et al., 2017), each trained network on one task is preserved and all networks are merged into one at the end of the sequence of tasks.
  Above approaches focus on adjusting the network weights. Another main approach is to add some data of past tasks to the new task training to prevent forgetting the past. Gradient Episodic Memory (GEM) (Lopez-Paz et al., 2017) stores a subset of training data for every finished task, and limits the loss function on these so-called ”memories”. Instead of keeping some real data from previous tasks, Generative Replay (GR) (Shin et al., 2017) keeps data generators for previous tasks and learns using a mix of real data of the new tasks and replayed data of previous tasks. Seff et al. (2017) proposed to solve continual generative modeling by combining the ideas of data generation and EWC.
  Other existing approaches include iCaRL (Rebuffi et al., 2017), Pathnet (Fernando et al., 2017), memory aware synapses (Aljundi et al., 2017), phantom sampling (Venkatesan et al., 2017), active long term memory networks (Furlanello et al., 2016), conceptor-aided backprop (He & Jaeger, 2018), gating networks (Masse et al., 2018; Serrà et al., 2018), dynamically expandable networks (DEN) (Yoon et al., 2018), progress & compress (Schwarz et al., 2018) (active column is distilled into the knowledge base, taking care to protect any previously acquired skills), and incremental regularized least squares (Camoriano et al., 2017). Most of those works suffer from accuracy deterioration as discussed in Section 1 while some dynamically expanding the network size (e.g., DEN and LWF), which results in a huge and complex model. Our method is different from these existing approaches.
  Several existing works generate parameters, but they are for different purposes. Denil et al. (2013) trained several different architectures by learning only a small number of weights and predicting the rest. Ha et al. (2016) used a small network to generate the weights for a larger network. Brock et al. (2017) learned an auxiliary HyperNet to generate weights for the main model. Our method uses the generated parameters for model adaptation for continual learning.
sections.heading: 5 CONCLUSION
sections.text: This paper proposed a novel approach PGMA to dealing with catastrophic forgetting. The approach learns to build a model with two sets of parameters. The first set is shared by all tasks learned so far and the second set is dynamically generated to adapt the model (solver) to suit each individual test example. As we discussed in related work, this is different from all existing methods. Experimental results showed that the proposed approach outperformed the existing baseline methods markedly.
sections.heading: ACKNOWLEDGMENTS
sections.text: We thank Zhangming Chan and Shen Gao for helping check the code. This work was supported by the National Key Research and Development Program of China (No. 2017YFC0804001). Bing Liu’s work was partially supported by National Science Foundation (NSF) under grant no. IIS 1838770, by a research contract with Huawei Technologies Co. Ltd., and by a research gift from Tencent Holdings Limited. Rui Yan’s work was supported by the National Science Foundation of China (NSFC No. 61672058; NSFC No. 61876196), CCF-Tencent Open Research Fund and Microsoft Research Asia (MSRA) Collaborative Research Program.
sections.heading: APPENDIX
sections.heading: A PROOF FOR QUESTION 1
sections.text: Proof: Our objective is to minimize ||R(x, θ∗i )−R(x, θ∗i−1)||. SinceR(·) is differentiable and its derivative function is bounded (as it uses 1-Lipschitz activation function, e.g., RELU or tanh), R(·) thus satisfies the Lipschitz condition. After some manipulation, we obtain:
  ||R(x, θ∗i )−R(x, θ∗i−1)|| = ||R(x, θ∗i )−R(x′, θ∗i ) +R(x′, θ∗i )−R(x′, θ∗i−1) +R(x′, θ∗i−1)−R(x, θ∗i−1)|| ≤ ||R(x, θ∗i )−R(x′, θ∗i )||+ ||R(x′, θ∗i )−R(x′, θ∗i−1)||+ ||R(x′, θ∗i−1)−R(x, θ∗i−1)|| ≤ L1||(x− x′)||+ ||R(x′, θ∗i )−R(x′, θ∗i−1)||+ L2||(x− x′)|| = ||R(x′, θ∗i )−R(x′, θ∗i−1)||︸ ︷︷ ︸
  minimize replayed samples
  +(L1 + L2)||(x− x′)||
  (10)
  where the last inference is based on Lipschitz continuity, and L1 and L2 are Lipschitz constants. We can see that if the solver has a small Lipschitz constant or the DG’s reconstruction error is small, minimizing Eq. 4 is consistent with constraining using the original data.
sections.heading: B PARAMETER SETTINGS
sections.heading: C MORE EXPERIMENTAL RESULTS
sections.heading: Memory
sections.text: • The memory used by our model is around 151.49MB, by IMM is around 38.96MB * task num (task num will be 3 if there are three tasks), and by GR is around 70.13MB. The memory required by all these systems are very small as compared to the total memory available in a modern computer.
  • Our data generator DG has around 1,460,361 parameters (around 44.57MB, calculated by 32 Byte per parameter), which is fixed. Saving data would need much more memory. Taking the MNIST dataset as an example. It has 60,000 training samples and the size of each sample is 28*28. The memory needed to store the data is 28*28*60,000 = 47,040,000 (around 358.89MB, calculated by 8 Byte per unit). And this number multiplies when the number of tasks increases.
sections.heading: Training and Test Time
sections.text: Taking the CIFAR10 dataset as an example (others are similar), the total time used by our model and baselines are shown in Table 6. We can see that our method needs a bit more time than baselines but not too much (under 25% for training and under 28% for testing compared with GR). This is a small price to pay for the major gain in accuracy and in dealing with the catastrophic forgetting problem.
  Accuracy on the Last Task
  ADAM gets the best accuracy on the last task as Adam optimizer learns the new task only and does not need to worry about the forgetting problem on the old tasks. For the last task, our PGMA system obtained comparable accuracy on average with the baselines designed for overcoming forgetting. However, we are much better at preventing forgetting of old tasks. The results for the last task are given in Table 7. Note that although EWC is quite good at the last task but it suffers seriously from forgetting as shown in Table 1 or Figure 3, which is consistent with results reported in other papers.
title: OVERCOMING CATASTROPHIC FORGETTING FOR CONTINUAL LEARNING VIA MODEL ADAPTATION