Regularization-based continual learning approaches generally prevent catastrophic forgetting by augmenting the training loss with an auxiliary objective. However in practical optimization scenarios with noisy data and/or gradients, it is possible that stochastic gradient descent can inadvertently change critical parameters. In this paper, we argue for the importance of regularizing optimization trajectories directly. We derive a new co-natural gradient update rule for continual learning whereby the new task gradients are preconditioned with the empirical Fisher information of previously learnt tasks. We show that using the conatural gradient systematically reduces forgetting in continual learning. Moreover, it helps combat overfitting when learning a new task in a low-resource scenario.
This paper amends the gradient update rule for continual learning using a natural-gradient-style formulation in order to regularise the trajectory during learning to forget previous task(s) less. They show experiments where this 'co-natural gradient' update rule improves some baselines. They also provide experiments showing the benefits of this update rule for low-resource finetuning settings.
Although the idea seems reasonable and interesting, I feel like this paper needs work both in the theory and experiments. Figure 1 is a nice visualisation of the key take-away point of the paper.
Theory: the authors take the natural-gradient updates from batch learning and just modify it so that the KL term is now for the previous task(s) instead of the current one. Although this may seem reasonable, I would appreciate some analysis as to what this implies or means.
Experiments: 
- Split CIFAR from Chaudhry et al. (2018b) uses 10 tasks, why does this paper use 20 tasks? 
- Previous works usually find that for EWC, large values of the \lambda hyperparameter provide best results. This corresponds to lower forgetting of previous tasks. The hyperparameter range in Appendix A.2.3 is only over small values of \lambda (by orders of magnitude). 
- Why do the authors only allow 1 epoch per task for Split CIFAR? This probably results in early stopping: the new tasks are not able to reach their new optimal points (with or without regularised trajectories). This seems to go against the intuition provided by Figure 1, where the authors are showing that changing the trajectory results in a better local minimum being found. 
In fact, by adding another regularisation term, it is unsurprising that co-natural gradient updates have less forgetting, as the extra regularisation term probably means the trained parameters are even closer to the previous parameters.
-------------------
EDIT: Score changed to 'Weak Accept' following discussion with the authors.