Learning neural networks with gradient descent over a long sequence of tasks is problematic as their fine-tuning to new tasks overwrites the network weights that are important for previous tasks. This leads to a poor performance on old tasks -a phenomenon framed as catastrophic forgetting. While early approaches use task rehearsal and growing networks that both limit the scalability of the task sequence orthogonal approaches build on regularization. Based on the Fisher information matrix (FIM) changes to parameters that are relevant to old tasks are penalized, which forces the task to be mapped into the available remaining capacity of the network. This requires to calculate the Hessian around a mode, which makes learning tractable. In this paper, we introduce Hessian-free curvature estimates as an alternative method to actually calculating the Hessian. In contrast to previous work, we exploit the fact that most regions in the loss surface are flat and hence only calculate a Hessian-vector-product around the surface that is relevant for the current task. Our experiments show that on a variety of well-known task sequences we either significantly outperform or are en par with previous work.
1. Summary:
The paper considers neural network training in the continual learning setting -- data arrive sequentially and we can not revisit past data. The paper proposes an approximate Laplace’s method, in which the Hessian the log likelihood of the data is approximated by some form of Hessian-vector project (? - I will get to this question mark below). The paper considers some benchmark continual learning datasets and compares the proposed approach to EWC and Kronecker-factored online Laplace. The performance of the proposed approach is similar to that of EWC and worse than Kronecker-factored Laplace in most cases. Another sales pitch that the paper brings up a lot is the low space complexity, however this benefit has not been fully demonstrated, given the small-scale network/experiments.
2. Opinion and rationales
I’m leaning towards “strong reject” as I think the presentation needs another round of polishing and that the technical contributions need to be clarified / unpacked. I explain my thinking below.
a. The presentation/explanation/flow are not clear.
The abstract does not read well. For example: “This requires to calculate the Hessian around a mode, which makes learning tractable. In this paper, we introduce Hessian-free curvature estimates as an alternative method to actually calculating the Hessian.” This sentence makes it sound like current approaches are tractable, so what this paper is trying to address? The technical summary is also not precise, the Hessian-free methods used in the paper is to compute Hessian-vector products, not the actual Hessian.
The introduction motivates the continual learning problem using generalisation of neural networks leading to the need for multi-task learning; however multi-task learning is not scalable given the large number of tasks and thus we need to learn sequentially. However, I find this motivation not clear: if multi-task learning and its scalability issue are the reasons why we need continual learning, with the scale of the experiments considered in the paper, wouldn’t it always more beneficial to use multi-task learning instead of continual learning?
The prior work section is also not clear, in my opinion. The paper starts out by describing EWC as Bayesian updates and cites MacKay (1992), then talks about the Kronecker-factored Laplace approximation as “address this shortcoming by adopting the Bayesian online learning approach”, as if these methods are very different while in fact, these methods are some variants of the Laplace approximation, with different ways to approximate the Hessian. The issues described in section 2.2 “two problems that stem from eq 1” are not very clear, for example, “without storing the information from all previous tasks there is no easy solution to update the posterior” (?). I would follow the presentation/explanation in Ritter et al (2018), Huszar (2018) [a note on the quadratic penalty of EWC] and section 5 of the variational continual learning paper (Nguyen et al 2018) to provide a more succinct connection between these methods.
The connections between this work and MAML in section 3 is not clear to me. The continual learning and meta learning settings are also quite different.
b. The technical contribution is not clear and if correct, if of limited novelty.
What is not clear from reading section 3 is what quantity is being approximated, at what point a Hessian-vector product appears and thus we can use Hessian-free methods to approximate it. The paper talks about flat loss surface and sampling a small subset of the Hessian -- I’m not sure I understand these connections. In eq 11, the paper replaces the Hessian values with results of the Hessian-vector-product approximations -- this seems very odd to me, especially in terms of semantics and units, Hessian and hessian-vector-products are two very different things. Again, it is perhaps just me not understanding what is being approximated in the first place. The technical contribution of this paper is thus limited: using Hessian-free methods to approximate Hessian-vector products in the continual learning context.
c. The performance of the proposed method is not super exciting. Pragmatically speaking, it is not clear why practitioners should be using this in the near future given Kronecker-factored Laplace works and scales well in practice and there are a plethora of other recent methods (e.g. VCL) that are also developed from the Bayesian principle and work much better than EWC.
3. Minor details:
a. In eq 1, the denominator should be p(D_{t+1} | D_{1:t}).
b. Figs 1 and 2, I would use the same colour scheme throughout to be consistent.