summaries: This paper proposes a model for continual learning. Specifically, it intends to merge parameter regularization methods and gradient projection methods. The regularization is achieved by taking a bayesian perspective over the distribution over parameters, that is approximated by a Laplace distribution in a variational fashion, with online updates. The gradient projection is addressed by trust-region optimization, resulting in an update rule that uses the precision matrix to precondition the gradient of the current task objective, plus a term that keeps the parameters close to the previous mean. Experiments are carried out for sequential problems on a stimulus-response dataset and on stroke MNIST.  The paper addresses the problem of catastrophic forgetting during sequential task learning with recurrent neural networks, a notoriously difficult continual learning setting.   The main theoretical contribution is a formalism, Natural Continual Learning (NCL), which generalizes several related continual learning approaches in the regularization-based family, including several recent SOTA techniques. This formalism applies to the probabilistic learning setting using neural networks, so it is not restricted to supervised learning or recurrent models.  Experimental evidence is provided to support the claim that NCL outperforms previous approaches.  Finally, interesting analogies are made to models of biological learning, which are also supported by accompanying experiments; similar analysis tools are applied to practical instantiations of the NCL framework in controlled experimental settings.  This paper proposes a novel regularization based continual learning approach, NCL, which is mainly used for recurrent neural network (RNN). Since NCL is based on Bayesian continual learning, NCL utilizes the approximate posterior of previous tasks as a prior for current task, and the Laplace approximation is used to compute the approximate posterior. Through simple toy analyses on convex and non-convex loss landscapes for NCL and other baseline methods, authors show that previous methods have a problem on the trajectories of convergence when learning a new task. Based on the results on simple toy analyses, authors proposed a novel trust region based regularization method that modulate the gradient by considering the curvature of prior distribution via its precision matrix. The proposed technique is highly similar to projection based methods, but its origin is quite different from previous methods. Experiment results show that NCL outperforms other baselines in stimulus-response and MNIST tasks.  The authors propose a method called Natural Continual Learning (NCL). NCL unifies the concepts of weight regularization and gradient projection into non-interfering subspaces. Gradient projection is implemented as a trust region optimization using the Fisher information matrix. The authors also introduce a Kronecker-factor approximation to improve the scalability properties of their method. The comparison against other continual learning approaches is conducted using recurrent neural network (RNN) architectures.  The authors introduce an extension to Bayesian continual learning whereby updates in successive tasks are constrained to be close to a “trust region” of the parameters found in preceding tasks. Exploiting the Bayesian nature of the model, the trust region in question can be defined in terms of the prior precision matrix. The authors show that the method, when applied to RNNs, is effective against baselines in a) a suite of neuroscience related tasks, and b) stroke MNIST datasets.
abstractText: Biological agents are known to learn many different tasks over the course of their lives, and to be able to revisit previous tasks and behaviors with little to no loss in performance. In contrast, artificial agents are prone to ‘catastrophic forgetting’ whereby performance on previous tasks deteriorates rapidly as new ones are acquired. This shortcoming has recently been addressed using methods that encourage parameters to stay close to those used for previous tasks. This can be done by (i) using specific parameter regularizers that map out suitable destinations in parameter space, or (ii) guiding the optimization journey by projecting gradients into subspaces that do not interfere with previous tasks. However, these methods often exhibit subpar performance in both feedforward and recurrent neural networks, with recurrent networks being of interest to the study of neural dynamics supporting biological continual learning. In this work, we propose Natural Continual Learning (NCL), a new method that unifies weight regularization and projected gradient descent. NCL uses Bayesian weight regularization to encourage good performance on all tasks at convergence and combines this with gradient projection using the prior precision, which prevents catastrophic forgetting during optimization. Our method outperforms both standard weight regularization techniques and projection based approaches when applied to continual learning problems in feedforward and recurrent networks. Finally, the trained networks evolve task-specific dynamics that are strongly preserved as new tasks are learned, similar to experimental findings in biological circuits.
authors.name: Ta-Chu Kao
authors.name: Kristopher T. Jensen
authors.name: Gido M. van de Ven
authors.name: Alberto Bernacchia
authors.name: Guillaume Hennequin
references.authors: R. Aljundi F. Babiloni M. Elhoseiny M. Rohrbach T. Tuytelaars
references.title: Memory aware synapses: Learning what (not) to forget
references.venue: In Proceedings of the European Conference on Computer Vision (ECCV),
references.authors: Amari S.-I
references.title: Natural gradient works efficiently in learning
references.venue: Neural Computation,
references.authors: K.C. Ames M.M. Churchland
references.title: Motor cortex signals for each arm are mixed across hemispheres and neurons yet partitioned within the population response. eLife, 8:e46159
references.authors: A. Bernacchia M. Lengyel G. Hennequin
references.title: Exact natural gradient in deep linear networks and its application to the nonlinear case
references.venue: Advances in Neural Information Processing Systems,
references.authors: A. Chaudhry P.K. Dokania T. Ajanthan P.H. Torr
references.title: Riemannian walk for incremental learning: Understanding forgetting and intransigence
references.venue: arXiv preprint arXiv:1801.10112
references.authors: C.A. Chestek A.P. Batista G. Santhanam M.Y. Byron A. Afshar J.P. Cunningham V. Gilja S.I. Ryu M.M. Churchland K.V. Shenoy
references.title: Single-neuron stability during repeated reaching in macaque premotor cortex
references.venue: Journal of Neuroscience,
references.authors: C. Clopath T. Bonhoeffer M. Hübener T. Rose
references.title: Variance and invariance of neuronal long-term representations
references.venue: Philosophical Transactions of the Royal Society B: Biological Sciences,
references.authors: Y. Cong M. Zhao J. Li S. Wang L. Carin
references.title: GAN memory with no forgetting
references.venue: Advances in Neural Information Processing Systems,
references.authors: E.D. de Jong
references.title: Incremental sequence learning. arXiv preprint arXiv:1611.03068
references.authors: M. De Lange T. Tuytelaars
references.title: Continual prototype evolution: Learning online from non-stationary data streams
references.venue: arXiv preprint arXiv:2009.00919
references.authors: A.K. Dhawale R. Poddar S.B. Wolff V.A. Normand E. Kopelowitz B.P. Ölveczky
references.title: Automated long-term recording and analysis of neural activity in behaving animals. eLife, 6:e27702
references.authors: L. Duncker L. Driscoll K.V. Shenoy M. Sahani D. Sussillo
references.title: Organizing recurrent network dynamics by task-computation to enable continual learning
references.venue: Advances in Neural Information Processing Systems,
references.authors: B. Ehret C. Henning M.R. Cervera A. Meulemans J. von Oswald B.F. Grewe
references.title: Continual learning in recurrent neural networks with hypernetworks
references.venue: arXiv preprint arXiv:2006.12109
references.authors: S.W. Failor M. Carandini K.D. Harris
references.title: Learning orthogonalizes visual cortical population
references.venue: codes. bioRxiv
references.authors: J.A. Gallego M.G. Perich R.H. Chowdhury S.A. Solla L.E. Miller
references.title: Longterm stability of cortical population dynamics underlying consistent behavior
references.venue: Nature Neuroscience,
references.authors: K. Ganguly J.M. Carmena
references.title: Emergence of a stable cortical map for neuroprosthetic control
references.venue: PLoS Biology,
references.authors: F. Huszár
references.title: On quadratic penalties in elastic weight consolidation
references.venue: arXiv preprint arXiv:1712.03847
references.authors: K. Jensen Kao T.-C M. Tripodi G. Hennequin
references.title: Manifold GPLVMs for discovering non-Euclidean latent structure in neural data
references.venue: Advances in Neural Information Processing Systems,
references.authors: K.T. Jensen N. Kadmon Harpaz A.K. Dhawale S.B.E. Wolff B.P. Ölveczky
references.title: Long-term stability of neural activity in the motor system. bioRxiv
references.authors: K.T. Jensen Kao T.-C J.T. Stone G. Hennequin
references.title: Scalable Bayesian GPFA with automatic relevance determination and discrete noise models. bioRxiv
references.authors: K.A. Katlowitz M.A. Picardo M.A. Long
references.title: Stable sequential activity underlying the maintenance of a precisely executed skilled behavior
references.authors: M.T. Kaufman M.M. Churchland S.I. Ryu K.V. Shenoy
references.title: Cortical activity in the null space: permitting preparation without movement
references.venue: Nature Neuroscience,
references.authors: D.P. Kingma J. Ba
references.title: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
references.authors: J. Kirkpatrick R. Pascanu N. Rabinowitz J. Veness G. Desjardins A.A. Rusu K. Milan J. Quan T. Ramalho A Grabska-Barwinska
references.title: Overcoming catastrophic forgetting in neural networks
references.venue: Proceedings of the National Academy of Sciences,
references.authors: F. Kunstner L. Balles P. Hennig
references.title: Limitations of the empirical fisher approximation for natural gradient descent
references.venue: arXiv preprint arXiv:1905.12558
references.authors: Z. Li D. Hoiem
references.title: Learning without forgetting
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
references.authors: N. Loo S. Swaroop R.E. Turner
references.title: Generalized variational continual learning. arXiv preprint arXiv:2011.12328
references.authors: H. Lütcke D.J. Margolis F. Helmchen
references.title: Steady or changing? long-term monitoring of neuronal population activity
references.venue: Trends in Neurosciences,
references.authors: V. Mante D. Sussillo K.V. Shenoy W.T. Newsome
references.title: Context-dependent computation by recurrent dynamics in prefrontal cortex
references.authors: J. Martens
references.title: New insights and perspectives on the natural gradient method
references.venue: arXiv preprint arXiv:1412.1193
references.authors: J. Martens R. Grosse
references.title: Optimizing neural networks with Kronecker-factored approximate curvature
references.venue: In ICML,
references.authors: C.V. Nguyen Y. Li T.D. Bui R.E. Turner
references.title: Variational continual learning
references.venue: arXiv preprint arXiv:1710.10628
references.authors: K. Osawa S. Swaroop A. Jain R. Eschenhagen R.E. Turner R. Yokota M.E. Khan
references.title: Practical deep learning with Bayesian principles
references.venue: arXiv preprint arXiv:1906.02506
references.authors: P. Pan S. Swaroop A. Immer R. Eschenhagen R.E. Turner M.E. Khan
references.title: Continual deep learning by functional regularisation of memorable past
references.venue: arXiv preprint arXiv:2004.14070
references.authors: A.J. Peters S.X. Chen T. Komiyama
references.title: Emergence of reproducible spatiotemporal activity during motor learning
references.venue: Nature, 510(7504):263–267
references.authors: Rebuffi S.-A A. Kolesnikov G. Sperl C.H. Lampert
references.title: icarl: Incremental classifier and representation learning
references.authors: H. Ritter A. Botev D. Barber
references.title: Online structured laplace approximations for overcoming catastrophic forgetting
references.venue: arXiv preprint arXiv:1805.07810
references.authors: D. Rolnick A. Ahuja J. Schwarz T. Lillicrap G. Wayne
references.title: Experience replay for continual learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: M.E. Rule T. O’Leary C.D. Harvey
references.title: Causes and consequences of representational drift
references.venue: Current Opinion in Neurobiology,
references.authors: G. Saha I. Garg K. Roy
references.title: Gradient projection memory for continual learning. arXiv preprint arXiv:2103.09762
references.authors: J. Schwarz W. Czarnecki J. Luketina A. Grabska-Barwinska Y.W. Teh R. Pascanu R. Hadsell
references.title: Progress & compress: A scalable framework for continual learning
references.venue: In International Conference on Machine Learning,
references.authors: H. Shin J.K. Lee J. Kim
references.title: Continual learning with deep generative replay
references.venue: arXiv preprint arXiv:1705.08690
references.authors: M.K. Titsias J. Schwarz Matthews A.G. d. G R. Pascanu Y.W. Teh
references.title: Functional regularisation for continual learning with gaussian processes
references.venue: In International Conference on Learning Representations
references.authors: H. Tseran M.E. Khan T. Harada T.D. Bui
references.title: Natural variational continual learning
references.venue: In Continual Learning Workshop NeurIPS,
references.authors: G.M. van de Ven H.T. Siegelmann A.S. Tolias
references.title: Brain-inspired replay for continual learning with artificial neural networks
references.venue: Nature Communications,
references.authors: G.M. van de Ven A.S. Tolias
references.title: Generative replay with feedback connections as a general strategy for continual learning
references.venue: arXiv preprint arXiv:1809.10635
references.authors: G.M. van de Ven A.S. Tolias
references.title: Three scenarios for continual learning
references.venue: arXiv preprint arXiv:1904.07734
references.authors: J. von Oswald C. Henning J. Sacramento B.F. Grewe
references.title: Continual learning with hypernetworks
references.venue: arXiv preprint arXiv:1906.00695
references.authors: T. Xu X. Yu A.J. Perlik W.F. Tobin J.A. Zweig K. Tennant T. Jones Y. Zuo
references.title: Rapid formation and selective stabilization of synapses for enduring motor
references.authors: G. Yang F. Pan Gan W.-B
references.title: Stably maintained dendritic spines are associated with lifelong memories
references.authors: G.R. Yang M.R. Joglekar H.F. Song W.T. Newsome Wang X.-J
references.title: Task representations in neural networks trained to perform many cognitive tasks
references.venue: Nature Neuroscience,
references.authors: B.M. Yu J.P. Cunningham G. Santhanam S.I. Ryu K.V. Shenoy M. Sahani
references.title: Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity
references.venue: Journal of Neurophysiology,
references.authors: G. Zeng Y. Chen B. Cui S. Yu
references.title: Continual learning of context-dependent processing in neural networks
references.venue: Nature Machine Intelligence,
references.authors: F. Zenke B. Poole S. Ganguli
references.title: Continual learning through synaptic intelligence
references.venue: In International Conference on Machine Learning,
sections.heading: 1 Introduction
sections.text: Catastrophic forgetting is a common feature of machine learning algorithms where training on a new task often leads to poor performance on previously learned tasks. This is in contrast to biological agents which are capable of learning many different behaviors over the course of their lives with little to no interference across tasks. The study of continual learning in biological networks may therefore help inspire novel approaches in machine learning, while the development and study of continual learning algorithms in artificial agents can help us better understand how this challenge is overcome in the biological domain. This is particularly true for more challenging continual learning settings where task identity is not provided at test time, and for continual learning in recurrent neural
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  networks (RNNs), which is important due to the practical and biological relevance of RNNs. However, continual learning in these settings has recently proven challenging for many existing algorithms, particularly those that rely on parameter regularization to mitigate forgetting [12, 13, 47]. In this work, we address these shortcomings by developing a continual learning algorithm that not only encourages good performance across tasks at convergence but also regularizes the optimization path itself using trust region optimization. This leads to improved performance compared to existing methods.
  Previous work has addressed the challenge of continual learning in artificial agents using weight regularization, where parameters important for previous tasks are regularized to stay close to their previous values [1, 17, 24, 32, 37, 54]. This approach can be motivated by findings in the neuroscience literature of increased stability for a subset of synapses after learning [49, 50]. More recently, approaches based on projecting gradients into subspaces orthogonal to those that are important for previous tasks have been developed in both feedforward [40, 53] and recurrent [12] neural networks. This is consistent with experimental findings that neural dynamics often occupy orthogonal subspaces across contexts in biological circuits [3, 14, 20, 22]. While these methods have been found to perform well in many continual learning settings, they also suffer from several shortcomings. In particular, while Bayesian weight regularization provides a natural way to weigh previous and current task information, this approach can fail in practice due to its approximate nature and often requires additional tuning of the importance of the prior beyond what would be expected in a rigorous Bayesian treatment [46]. In contrast, while projection-based methods have been found empirically to mitigate catastrophic forgetting, it is unclear how the ‘important subspaces’ should be selected and how such methods behave when task demands begin to saturate the network capacity.
  In this work, we develop natural continual learning (NCL), a new method that combines (i) Bayesian continual learning using weight regularization with (ii) an optimization procedure that relies on a trust region constructed from an approximate posterior distribution over the parameters given previous tasks. This encourages parameter updates predominantly in the null-space of previously acquired tasks while maintaining convergence to a maximum of the Bayesian approximate posterior. We show that NCL outperforms previous continual learning algorithms in both feedforward and recurrent networks. We also show that the projection-based methods introduced by Duncker et al. [12] and Zeng et al. [53] can be viewed as approximations to such trust region optimization using the posterior from previous tasks. Finally, we use tools from the neuroscience literature to investigate how the learned networks overcome the challenge of continual learning. Here, we find that the networks learn latent task representations that are stable over time after initial task learning, consistent with results from biological circuits.
sections.heading: 2 Method
sections.text: Notations We use X>, X−1, Tr(X) and vec(X) to denote the transpose, inverse, trace, and column-wise vectorization of a matrix X . We use X ⊗ Y to represent the Kronecker product between matrices X ∈ Rn×n and Y ∈ Rm×m such that (X ⊗ Y )mi+k,mj+l = XijYkl. We use bold lower-case letters x to denote column vectors. Dk refers to a ‘dataset’ corresponding to task k, which in this work generally consists of a set of input-output pairs {x(i)k ,y (i) k } such that
  `k(θ) := log p(Dk|θ) = ∑ i log pθ(y (i) k |x (i) k ) is the task-related performance on task k for a model with parameters θ. Finally, we use D̂k to refer to a dataset generated by inputs from the kth task where {ŷ(i)k ∼ pθ(y|x (i) k )} are drawn from the model distributionM.
sections.heading: 2.1 Bayesian continual learning
sections.text: Problem statement In continual learning, we train a model on a set of K tasks {D1, . . . ,DK} that arrive sequentially, where the data distribution Dk for task k in general differs from D6=k. The aim is to learn a probabilistic model p(D|θ) that performs well on all tasks. The challenge in the continual learning setting stems from the sequential nature of learning, and in particular from the common assumption that the learner does not have access to “past” tasks (i.e., Dj for j < k) when learning task k. While we enforce this stringent condition in this paper, our approach may be easily combined with memory-based techniques such as coresets or generative replay [8, 10, 13, 32, 34, 36, 38, 42, 43, 45, 48].
  Bayesian approach The continual learning problem is naturally formalized in a Bayesian framework whereby the posterior after k−1 tasks is used as a prior for task k. More specifically, we choose a prior p(θ) on the model parameters and compute the posterior after observing k tasks according to Bayes’ rule:
  p(θ|D1:k) ∝ p(θ) k∏
  k′=1
  p(Dk′ |θ)
  ∝ p(θ|D1:k−1)p(Dk|θ), (1)
  where D1:k is a concatenation of the first k tasks (D1, . . . ,Dk). In theory, it is thus possible to compute the exact posterior p(θ|D1:k) after k tasks, while only observing Dk, by using the posterior p(θ|D1:k−1) after k − 1 tasks as a prior. However, as is often the case in Bayesian inference, the difficulty here is that the posterior is typically intractable. To address this challenge, it is common to perform approximate online Bayesian inference. That is, the posterior p(θ|D1:k−1) is approximated by a parametric distribution with parameters φk−1. The approximate posterior q(θ;φk−1) is then used as a prior for task k.
  Online Laplace approximation A common approach is to use the Laplace approximation whereby the posterior p(θ|D1:k−1) is approximated as a multivariate Gaussian q using local gradient information [17, 24, 37]. This involves (i) finding a mode µk of the posterior during task k, and (ii) performing a second-order Taylor expansion around µk to construct an approximate Gaussian posterior q(θ;φk) = N (θ;µk,Λ−1k ), where Λk is the precision matrix and φk = (µk,Λk). In this case, gradient-based optimization is used to find the posterior mode on task k (c.f. Equation 1):
  µk = arg max θ
  log p(θ|Dk,φk−1) (2)
  = arg max θ
  log p(Dk|θ) + log q(θ;φk−1) (3)
  = arg max θ
  `k(θ)− 1
  2 (θ − µk−1)>Λk−1(θ − µk−1)︸ ︷︷ ︸
  := Lk(θ)
  (4)
  The precision matrix Λk is given by the Hessian of the negative log posterior at µk: Λk = − ∇2θ log p(θ|Dk,φk−1) ∣∣ θ=µk = H(Dk,µk) + Λk−1, (5)
  where H(Dk,µk) = − ∇2θ log p(Dk|θ) ∣∣ θ=µk is the Hessian of the negative log likelihood of Dk.
  Continual learning with the online Laplace approximation thus involves two steps for each new task Dk. First, given Dk and the previous posterior q(θ;µk−1,Λ−1k−1) (i.e. the new prior), µk is found using gradient-based optimization (Equation 4). This step can be interpreted as optimizing the likelihood of Dk while penalizing changes in the parameters θ according to their importance for previous tasks, as determined by the prior precision matrix Λk−1. Second, the new posterior precision matrix Λk is computed according to Equation 5.
  Approximating the Hessian In practice, computing Λk presents two major difficulties. First, because q(θ;φk) is a Gaussian distribution, Λk has to be positive semi-definite (PSD), which is not guaranteed for the Hessian H(Dk,µk). Second, if the number of model parameters nθ is large, it may be prohibitive to compute a full (nθ × nθ) matrix. To address the first issue, it is common to approximate the Hessian with the Fisher information matrix (FIM; 17, 30, 37):
  Fk = Ep(D̂k|θ) [ ∇θ log p(D̂k|θ)∇θ log p(D̂k|θ)> ]∣∣∣ θ=µk ≈ H(Dk,µk) (6)
  The FIM is PSD, which ensures that Λk = ∑k k′=1 Fk′ is also PSD. Computing Fk may still be impractical if there are many model parameters, and it is therefore common to further approximate the FIM using structured approximations with fewer parameters. In particular, a diagonal approximation to Fk recovers Elastic Weight Consolidation (EWC; 24), while a Kronecker-factored approximation [31] recovers the method proposed by Ritter et al. [37]. We denote this method ‘KFAC’ and use it in Section 3 as a comparison for our own Kronecker-factored method.
sections.heading: 2.2 Natural continual learning
sections.text: While the online Laplace approximation has been applied successfully in several continual learning settings [24, 37], it has also been found to perform sub-optimally on a range of problems [12, 46]. Additionally, its Bayesian interpretation in theory prescribes a unique way of weighting the contributions of previous and current tasks to the loss. However, to perform well in practice, weight regularization approaches have been found to require ad-hoc re-weighting of the prior term by several orders of magnitude [24, 37, 46]. These shortcomings could be due to an inadequacy of the approximations used to construct the posterior (Section 2.1). However, we show in Figure 1 that standard gradient descent on the Laplace posterior has important drawbacks even in the exact case. First, we show that exact Bayesian inference on a simple continual regression problem can produce indirect optimization paths along which previous tasks are transiently forgotten as a new task is being learned (Figure 1A; green). Second, when the loss is non-convex, we show that exact Bayesian inference can still lead to catastrophic forgetting (Figure 1B; green).
  An alternative approach that has found recent success in a continual learning setting involves projection based methods which restrict parameter updates to a subspace that does not interfere with previous tasks [12, 53]. However, it is not immediately obvious how this projected subspace should be selected in a way that appropriately balances learning on previous and current tasks. Additionally, such projection-based algorithms have fixed points that are minima of the current task, but not necessarily minima of the (negative) Bayesian posterior. This can lead to catastrophic forgetting in the limit of long training times (Figure 1; pink), unless the learning rate is exactly zero in directions that interfere with previous tasks.
  To combine the desirable features of both classes of methods, we introduce “Natural Continual Learning” (NCL) – an extension of the online Laplace approximation that also restricts parameter updates to directions which do not interfere strongly with previous tasks. In a Bayesian setting, we can conveniently express what is meant by such directions in terms of the prior precision matrix Λ. In particular, ‘flat’ directions of the prior (low precision) correspond to directions that will not significantly affect the performance on previous tasks. Formally, we derive NCL as the solution of a trust region optimization problem. This involves minimizing the posterior loss Lk(θ) within a region of radius r centered around θ with a distance metric of the form d(θ,θ + δ) = √ δ>Λk−1δ/2 that takes into account the curvature of the prior via its precision matrix Λk−1:
  δ = arg min δ
  Lk(θ) +∇θLk(θ)>δ subject to 1
  2 δ>Λk−1δ ≤ r2, (7)
  where Lk(θ + δ) ≈ Lk(θ) + ∇θLk(θ)>δ is a first-order approximation to the updated Laplace objective. The solution to this subproblem is given by δ ∝ Λ−1k−1∇θ`k(θ) − (θ − µk−1) (see
  Appendix A for a derivation), which gives rise to the NCL update rule θ ← θ + γ [ Λ−1k−1∇θ`k(θ)− (θ − µk−1) ] (8)
  for a learning rate parameter γ (which is implicitly a function of r in Equation 7). To get some intuition for this learning rule, we note that Λ−1k−1 acts as a preconditioner for the first (likelihood) term, which drives learning on the current task while encouraging parameter changes predominantly in directions that do not interfere with previous tasks. Meanwhile, the second term encourages θ to stay close to µk−1, the optimal parameters for the previous task. As we illustrate in Figure 1, this combines the desirable features of both Bayesian weight regularization and projection-based methods. In particular, NCL shares the fixed points of the Bayesian posterior while also mitigating intermediate or complete forgetting of previous tasks by preconditioning with the prior covariance. Notably, if the loss landscape is non-convex (as it generally will be), NCL can converge to a different local optimum from standard weight regularization despite having the same fixed points (Figure 1B).
  Implementation The general NCL framework can be applied with different approximations to the Fisher matrix Fk in Equation 6 (see Section 2.1). In this work, we use a Kronecker-factored approximation [31, 37]. However, even after making a Kronecker-factored approximation to Fk for each task k, it remains difficult to compute the inverse of a sum of k Kronecker products (c.f. Equation 5). To address this challenge, we derived an efficient algorithm for making a Kroneckerfactored approximation to Λk = Fk + Λk−1 ≈ Ak ⊗Gk when Λk−1 = Ak−1 ⊗Gk−1 and Fk are also Kronecker products. This approximation minimizes the KL-divergence between N (µk, (Ak ⊗ Gk)
  −1) and N (µk, (Λk−1 + Fk)−1) (see Appendix G for details). Before training on the first task, we assume a spherical Gaussian prior θ ∼ N (0, p−2w I). The scale parameter pw can either be set to a fixed value (e.g. 1) or treated as a hyperparameter, and we optimize pw explicitly for our experiments in feedforward networks. NCL also has a parameter α which is used to stabilize the matrix inversion Λ−1k−1 ≈ (Ak−1 ⊗Gk−1 + α2I)−1 (Appendix E). This is equivalent to a hyperparameter used for such matrix inversions in OWM [53] and DOWM [12], and it is important for good performance with these methods. The pw and α are largely redundant for NCL, and we generally prefer to fix α to a small value (10−10) and optimize the pw only. However, for our experiments in RNNs, we instead fix pw = 1 and perform a hyperparameter optimization over α for a more direct comparison with OWM and DOWM. The NCL algorithm is described in pseudocode in Appendix E together with additional implementation and computational details. Finally, while we have derived NCL with a Laplace approximation in this section for simplicity, it can similarly be applied in the variational continual learning framework of Nguyen et al. [32] (Appendix J). Our code is available online1.
sections.heading: 2.3 Related work
sections.text: As discussed in Section 2.1, our method is derived from prior work that relies on Bayesian inference to perform weight regularization for continual learning [17, 24, 32, 37]. However, we also take inspiration from the literature on natural gradient descent [2, 25] to introduce a preconditioner that encourages parameter updates primarily in flat directions of previously learned tasks (Appendix H).
  Recent projection-based methods [12, 40, 53] have addressed the continual learning problem using an update rule of the form θ ← θ + γPL∇θ`k(θ)PR, (9) wherePL andPR are projection matrices constructed from previous tasks which encourage parameter updates that do not interfere with performance on these tasks. Using Kronecker identities, we can rewrite Equation 9 as θ ← θ + γ(PR ⊗ PL)∇θ`k(θ). (10) This resembles the NCL update rule in Equation 8 where we identify PR ⊗PL with the approximate inverse prior precision matrix used for gradient preconditioning in NCL, Λ−1k−1 = A −1 k−1 ⊗G −1 k−1. Indeed, we note that for a Kronecker-structured approximation to Fk, the matrixAk−1 approximates the empirical covariance matrix of the network activations experienced during all tasks up to k − 1 (4, 31, Appendix D), which is exactly the inverse of the projection matrix PR used in previous work [12, 53]. We thus see that NCL takes the form of recent projection-based continual learning algorithms with two notable differences:
  1https://github.com//tachukao/ncl
  (i) NCL uses a left projection matrix PL designed to approximate the posterior covariance of previous tasks Λ−1k−1 ≈ PR ⊗PL (i.e., the prior covariance on task k; Appendix D), while Zeng et al. [53] use the identity matrix I and Duncker et al. [12] use the covariance of recurrent inputs (Appendix F). Notably, both of these choices of PL still provide reasonable approximations to Λ−1k−1, and thus the parameter updates of OWM and DOWM can also be viewed as projecting out steep directions of the prior on task k (Appendix F). (ii) NCL includes an additional regularization term (θ − µk−1) derived from the Bayesian posterior objective, while Duncker et al. [12] and Zeng et al. [53] do not use such regularization. Importantly, this means that while NCL has a similar preconditioner and optimization path to these projection based methods, NCL has stationary points at the modes of the approximate Bayesian posterior while the stationary points of OWM and DOWM do not incorporate prior information from previous tasks (c.f. Figure 1).
  It is also interesting to note that previous Bayesian continual learning algorithms include a hyperparameter λ that scales the prior compared to the likelihood term for the current task [27]:
  L(λ)k (θ) = log p(Dk|θ)− λ(θ − µk−1) >Λk−1(θ − µk−1). (11)
  To minimize this loss and thus find a mode of the approximate posterior, it is common to employ pseudo-second-order stochastic gradient-based optimization algorithms such as Adam [23] that use their own gradient preconditioner based on an approximation to the Hessian of Equation 11. Interestingly, this Hessian is given by Hk = −H(Dk,θ) − λΛk−1, which in the limit of large λ becomes increasingly similar to preconditioning with the prior precision as in NCL. Consistent with this, previous work using the online Laplace approximation has found that large values of λ are generally required for good performance [24, 37, 46]. Recent work has also combined Bayesian continual learning with natural gradient descent [33, 44], and in this case a relatively high value of λ = 100 was similarly found to maximize performance [33].
sections.heading: 3 Experiments and results
sections.heading: 3.1 NCL in feedforward networks
sections.text: To verify the utility of NCL for continual learning, we first compared our algorithm to standard methods in feedforward networks across two continual learning benchmarks: split MNIST and split CIFAR-100 (see Appendix B for task details). For each benchmark, we considered three continual learning settings [47]. In the ‘task-incremental’ setting, task identity is available to the network at test time, in our case via a multi-head output layer [5]. In the ‘domain-incremental’ setting, task identity is unavailable at test time, and the output layer is shared between all tasks. Finally, in the ‘class-incremental’ setting, the network has to both infer task identity and solve the task, in our case by performing classification over all possible classes irrespective of which task the input in question is drawn from.
  van de Ven and Tolias previously showed that parameter regularization methods such as EWC perform poorly in the domain- and class-incremental settings [47]. We therefore applied NCL as well as synaptic intelligence [SI; 54], online EWC [41], Kronecker factored EWC [KFAC; 37], and orthogonal weight modification [OWM; 53] to split MNIST and split CIFAR-100 in the task-, domain- and class-incremental learning settings. For these continual learning problems, we found that NCL outperformed all the baseline methods in the task- and domain-incremental learning settings (Figure 2). In the class-incremental settings, we found that NCL performed comparably to but slightly worse than OWM. However, both OWM and NCL comfortably outperformed the other compared methods in this setting. These results suggest that the subpar performance of parameter regularization methods can be alleviated by regularizing their optimization paths, particularly in the domain- and class-incremental learning settings.
  For the split MNIST and split CIFAR-100 experiments, each baseline method had a single hyperparameter (c for SI, λ for EWC and KFAC, α for OWM, and pw for NCL; Appendix E) that was optimized on a held-out seed (see Appendix I.2). However, by setting the NCL prior to a unit Gaussian, we were also able to achieve good performance across task sets in a hyperparameter-free setting, further highlighting the robustness of the method (see “NCL (no opt)” in Figure 2).
sections.heading: 3.2 NCL in recurrent neural networks
sections.text: We then proceeded to consider how NCL compares to previous methods in recurrent neural networks (RNNs), a setting that has recently proven challenging for continual learning [12, 13] and which is of interest to the study of continual learning in biological circuits [12, 51]. In these experiments, the task identity is available to the RNN (i.e., we consider the task-incremental learning setting).
  Stimulus-response tasks In this section, we consider a set of neuroscience inspired ‘stimulusresponse’ (SR) tasks (51; details in Appendix B). We first compared the performance and behavior of NCL to OWM, the top performing method in the feedforward setting (Figure 2), and to the projection-based DOWM method designed explicitly for RNNs [12]. For a more direct comparison with OWM and DOWM, we fixed the NCL prior to a unit Gaussian for all RNN experiments and instead performed a hyperparameter optimization over ‘α’ used to regularize the matrix inversions for all three methods (Section 2.2, Appendix E, Appendix I.2, 12, 53). Following previous work, we trained RNNs with 256 recurrent units to sequentially solve six stimulus-response tasks [12, 50]. While NCL, OWM and DOWM all managed to learn the six tasks without catastrophic forgetting, we found that NCL achieved superior average performance across tasks after training (Figure 3A).
  We then compared NCL, OWM, and DOWM to KFAC, the top performing parameter regularization method in our feedforward experiments (Figure 2) which uses Adam [23] to optimize the objective in Equation 4 with a Kronecker-factored approximation to the posterior precision matrix (Section 2.1; 37). Consistent with the results shown in Duncker et al. [12], we found that NCL, OWM, and DOWM outperformed KFAC with λ = 1 (Figure 3A; see also 12 for a comparison of DOWM and EWC). We note that NCL and KFAC optimize the same objective function (Equation 4) and approximate the posterior precision matrix in the same way, but they differ in the way they precondition the gradient of the objective. These results thus demonstrate empirically that the choice of optimization algorithm is important to prevent forgetting, consistent with the intuition provided by Figure 1.
  In feedforward networks, poor performance with weight regularization approaches such as EWC and KFAC has been mitigated by optimizing the hyperparameter λ, which increases the importance of the prior term compared to a standard Bayesian treatment (Equation 11; Section 3.1, 24, 27, 37). We confirmed this here by performing a grid search over λ, which showed that KFAC with λ ∈
  [100, 1000] could perform comparably to the projection-based methods (Appendix I.1; Figure 3A). We hypothesize that the good performance provided by high λ is partly due to the approximate second order nature of Adam which, together with the relative increase in the prior term compared to the data term, leads to preconditioning with a matrix resembling the prior Λk−1 (Section 2.3). In support of this hypothesis, we found that the KL divergence between the Adam preconditioner and the approximate prior precision Λk−1 decreased with increasing λ, and that the performance of KFAC with Adam could also be rescued by increasing λ only when computing the preconditioner while retaining λ = 1 when computing the gradients (Appendix I.1).
  Stroke MNIST One way to challenge the continual learning algorithms further is to increase the number of tasks. We thus considered an augmented version of the stroke MNIST dataset [SMNIST; 9]. The original dataset consists of the MNIST digits transformed into pen strokes with the direction of the stroke at each time point provided as an input to the network. Similar to Ehret et al. [13], we constructed a continual learning problem by considering consecutive binary classification tasks inspired by the split MNIST task set. We further increased the number of tasks by including a set of extra digits where the x and y dimensions have been swapped in the input stroke data, and another set where both the x and y dimensions have changed sign. We also added high-variance noise to the inputs to increase the task difficulty. This gave rise to a total of 15 binary classification tasks, each with unique digits not used in other tasks, which we sought to learn in a continual fashion using an RNN with 30 recurrent units (see Appendix B for details).
  As for the SR task set in Section 3.2, we found that NCL outperformed previous projection-based methods (Figure 3B). We again found that weight regularization with a KFAC approximation performed poorly with λ = 1, and that this poor performance could be partially rescued by optimizing over λ (Figure 3B). To investigate how the difference in performance between NCL and DOWM was affected by their different approximations to the Fisher matrix (Appendix F), we implemented NCL using the DOWM projection matrices as an alternative approximation to the inverse Fisher matrix. We refer to this method as Laplace-DOWM. We then considered how the performance on each task at the end of training depended on task number, averaged over different task permutations (Figure 3C). We found that while Laplace-DOWM outperformed NCL on the first task, this method generally performed worse on subsequent tasks. Notably, Laplace-DOWM exhibited a near-monotonic decrease in relative performance with task number, which is consistent with the intuition that DOWM overestimates the dimensionality of the parameter subspace that matters for previous tasks (Appendix F). In contrast, although neural circuits are known to use orthogonal subspaces in different contexts, there is no general sense that learning more tasks in the past should systematically hinder learning in future contexts for biological agents.
sections.heading: 3.3 Dissecting the dynamics of networks trained on the SMNIST task set
sections.text: To further investigate how the trained RNNs solve the continual learning problems and how this relates to the neuroscience literature, we dissected the dynamics of networks trained on the SMNIST task set using the NCL algorithm. To do this, we analyzed latent representations of the RNN activity trajectories, as is commonly done to study the collective dynamics of artificial and biological networks [15, 18, 20, 29, 52]. We considered two consecutive classification tasks, namely classifying 4’s vs 5’s (k = 2) and classifying 1’s vs 7’s (k = 3). For each of these tasks, we trained a factor analysis model right after the task was learned, using network activity collected while presenting 50 examples of each of the two input digits associated with the task. We then tracked the network responses to the same set of stimuli at various stages of learning, both before and after the task in question was acquired, using the trained factor analysis model to visualize low-dimensional summaries of the dynamics (Figure 4).
  Consistent with the network having successfully learned to solve these two tasks, we found that latent trajectories diverged over time for the two types of inputs in each task. Critically, these diverging dynamics only emerged after the task was learned, and remained highly stable thereafter (Figure 4). The stability of the task-associated representations is consistent with recent work in the neuroscience literature showing that, in a primate reaching task, latent neural trajectories remain stable after learning [15]. Since here we have access to the activity of all neurons throughout the task, we proceeded to quantify the source of this stability at the level of single units. The stability of such single-neuron dynamics after learning has recently been a topic of much interest in biological circuits [7, 28, 39]. In the RNNs, we found that the single-unit representations of a given digit changed during learning of the task involving that digit but stabilized after learning, consistent with work in several distinct biological circuits [6, 11, 16, 19, 21, 35]. Similar results were found using the DOWM algorithm, which was explicitly designed to preserve network dynamics on previously learned tasks [12]. Interestingly, the stable task representations learned by NCL and DOWM differed markedly from a network trained with replay for continual learning, which instead led to task representations that continued to change after initial task acquisition (Appendix I.4). This illustrates how different approaches to continual learning can lead to qualitatively different circuit dynamics, and it suggests the use of continual learning in artificial networks as a model system for biological continual learning.
sections.heading: 4 Discussion
sections.text: In summary, we have developed a new framework for continual learning based on approximate Bayesian inference combined with trust-region optimization. We showed that this framework encompasses recent projection-based methods and found that it performs better than naive weight regularization. This was particularly evident when task identity was not provided at test time and in recurrent neural networks, settings which have previously been challenging for many continual learning algorithms [12, 13, 47]. Furthermore, we showed that our principled probabilistic approach outperforms previous projection-based methods [12, 53], in particular when the number of tasks and their complexity challenges the network’s capacity. Finally, we analyzed the dynamics of the learned RNNs in a sequential binary classification problem, where we found that the latent dynamics adapt to each new task. We also found that the task-associated dynamics were subsequently conserved during further learning, consistent with experimental reports of stable neural representations [11, 15, 19]. Importantly, our results suggest that preconditioning with the prior covariance can lead to improved performance over existing continual learning algorithms. In future work, it will therefore be interesting to apply this idea to other weight regularization approaches such as EWC with a diagonal approximate posterior [24]. Finally, a separate branch of continual learning utilizes replay-like mechanisms to reduce catastrophic forgetting [8, 26, 34, 42, 43, 46]. While our work has focused on weight regularization, such regularization and replay are not mutually exclusive. Instead, these two approaches have been found to further improve robustness to catastrophic forgetting when combined [32, 45].
  Impact and limitations While we have shown that NCL represents an important conceptual and methodological advance for continual learning, it also comes with several limitations. One such limitation arises from the relative difficulty of computing the prior Fisher matrix which is needed for our projection step. Indeed the success of methods such as Adam [23] and EWC [24] is due in part to their ease of implementation which facilitates broad applicability. It will therefore be interesting to investigate how approximations such as a running average of a diagonal approximation to the empirical Fisher matrix as used in Adam could facilitate the development of simple yet powerful variants of NCL.
  Furthermore, while NCL mitigates the need to overcount the prior from previous tasks via λ as in KFAC, it does introduce two other (largely redundant) hyperparameters in the form of (i) the scale of the prior before the first task, and (ii) the parameter α used to regularize the inversion of the prior Fisher matrix, similar to OWM and DOWM [12, 53]. While α is an important hyperparameter for OWM and DOWM and we also optimize it in the RNN setting for a more direct comparison (Section 3.2), we find it more natural to set this parameter to a constant small value present only for numerical stability (Appendix E). This leaves the prior scale which we optimize explicitly in the feedforward setting (Section 3.1). However, in future work it would be interesting to consider whether a good prior can be determined in a data free manner to make NCL a hyperparameter-free method. Finally, computing the Fisher matrix used for pre-conditioning requires explicit knowledge of task boundaries. In future work, it will therefore be interesting to develop an algorithm similar to NCL which also works for online learning problems with continually changing task distributions.
  Addressing these challenges is important since machine learning algorithms increasingly need to be robust to changing data distributions and dynamic task specifications as they become more prominent in our everyday lives. Much work has therefore gone into the development of methods for continual learning in the machine learning community. However, with the increasing prevalence of practical algorithms for continual learning, it also becomes increasingly important that we understand how and why these algorithms work – insights that can also help us understand when they might fail. In this work, we have therefore attempted to shed light on the relationship between recent methods for continual learning as well as developing a new algorithm with a principled probabilistic interpretation that makes its underlying assumptions more explicit. Taken together, we hope that this work will help improve our understanding of methods for continual learning while also providing an avenue for further research to increase the reliability and robustness of future continual learning algorithms.
sections.heading: Acknowledgements
sections.text: We are grateful to Siddharth Swaroop, Lea Duncker, Laura Driscoll, Naama Kadmon Harpaz, and Yashar Ahmadian for insightful discussions. We thank Siddharth Swaroop and Robert Pinsler for useful comments on the manuscript.
  Funding disclosure
  KTJ was funded by a Gates Cambridge scholarship. GMV was supported by the Lifelong Learning Machines (L2M) program of the Defence Advanced Research Projects Agency (DARPA) via contract number HR0011-18-2-0025.
title: Natural continual learning: success is a journey, not (just) a destination