In continual learning, the learner faces a stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. To address such catastrophic forgetting, many continual learning methods implement different types of experience replay, re-learning on past data stored in a small buffer known as episodic memory. In this work, we complement experience replay with a meta-learning technique that we call "anchoring": the learner updates its knowledge on the current task, while keeping predictions on some anchor points of past tasks intact. These anchor points are learned using gradient-based optimization as to maximize forgetting of the current task, in hindsight, when the learner is fine-tuned on the episodic memory of past tasks. Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the standard experience replay in terms of both accuracy and forgetting metrics and for various sizes of episodic memories.
This paper proposes an approach for continual learning that improves existing memory / replay -based methods, by learning anchors for each previous task. This is achieved by computing a temporary parameter update with a conventional loss (on both the current task and samples from an episodic memory buffer); and trading off this update with an additional loss measuring the reduction in performance on a set of "anchor samples" from previous tasks. Gradient descent is used to estimate the task-specific anchors that are most crucial across all (past and future) tasks.
The paper is well-written, it is a novel and well-explained idea, and the results are compelling. I lean towards acceptance, but I think there are a number of things that could be improved before final publication.
1) On how the anchors are estimated:
a) It is stated on page 3 that "... choose each e_t as a tool to minimize the Forgetting metric (Eq. 3). That is, we are interested in letting e_t be the example from task t that would maximize the amount of forgetting about task t throughout the entire learning experience...."
This appears to be contradictory - is the aim to find the example that would maximise the amount of forgetting for a given task after training on all tasks, in order to keep samples that are most "crucial"? 
This section could be clearer, and further discussion and intuition would be beneficial for the reader.
b) Does it always make sense to choose anchors that maximise forgetting on the current task? Is this akin to finding examples that would optimally alter the decision boundary, in a similar fashion to support vectors?
c) I really like the intuition behind optimizing for the anchors via gradient descent (since finding examples that would be most forgotten in future violates the continual learning assumption). It would be interesting to visualize what anchors are found. For example, do they tend to be close to the centroid of a class (such that the mean embedding regulariser is low)? Are they examples that tend to be visually close to other classes, or are they outliers?
d) Given that anchors are already estimated and used to alleviate forgetting, what's the benefit of the additional episodic memory? An ablation is performed with different memory sizes, but what if *only* anchors are used (ie. a size of zero)? This would be useful in delineating the benefits of the anchors versus standard replay.
2) On the two-step optimisation (Eqn. 5):
a) It's not clear to me why this constitutes a meta-learning process: there is no adaptation at test time, nor does this perform task inference or learn how to learn. In this case, the method just uses a single gradient step to compute the change in predictions at the anchor points.
b) I wonder if there is a relationship to the loss used in Riemer et al, ICLR 2019. In particular, it seems that the gradient dot product term (used in that work to maximise transfer) may be related to a first-order Taylor expansion of the L2 term in this work (the change in predictions at anchor points, Eqn 5). It would be good to clarify whether this is the case, and add further discussion and intuition as needed.
3) On the relationship to existing work:
a) The approach seems conceptually most similar to iCARL (cited in the paper), which maintains "exemplars" from each previous task in order to avoid forgetting. I think the writing could be much more explicit about this, and both (i) compare against the performance of iCARL, and (ii) clearly discuss the differences between the iCARL exemplars and the anchors obtained in this work.
The equivalent result for splitCIFAR in the iCARL paper (for 20 tasks x 5 classes) appears to be around 45-50%, so I think it is necessary to compare and evaluate the source of this large improvement in performance.
b) Another paper worth discussing and comparing further is Variational Continual Learning - which is effectively a regularization-based method augmented with a coreset / episodic memory.
c) The paper seems quite centred on memory/replay- based approaches to continual learning; the abstract in particular says "SOTA continual learning methods implement different types of experience replay", but this fails to focus on non- memory-based methods. I would temper the claim to "many approaches to continual learning...", and spread the focus beyond replay-based approaches.