In a continual learning setting, new categories may be introduced over time, and an ideal learning system should perform well on both the original categories and the new categories. While deep neural nets have achieved resounding success in the classical setting, they are known to forget about knowledge acquired in prior episodes of learning if the examples encountered in the current episode of learning are drastically different from those encountered in prior episodes. This makes deep neural nets ill-suited to continual learning. In this paper, we propose a new model that can both leverage the expressive power of deep neural nets and is resilient to forgetting when new categories are introduced. We demonstrate an improvement in terms of accuracy on original classes compared to a vanilla deep neural net.
This paper considers the use of a metric learning approach in a continual/lifelong classification settings. Experiments show in the case of two tasks forgetting can be minimized by using the approach. 
Methods
The proposed method appears to be a standard triplet loss. The authors add a second term to the triplet loss that is essentially making the loss a combination of the triplet and siamese loss. It’s not really explained anywhere why they do this and whether its essential to the performance. 
Is there anything specific to continual learning done or is the paper essentially pointing out this existing method (metric learning + nearest neighbor) is surprisingly effective for forgetting. If this is the case the authors should present it in this way I think. 
Although triplet loss can often yield reasonably performance on classification problems it tends to not perform as well as cross entropy loss, this is observed in other works as well as this one.
A major question of mine: it is not clear from the method nor experiments what samples are stored after task A for the kNN classifier. Is it all of the data samples from the previous task? 
Experiments
The experimental results consider a custom continual learning setup where there is two sets of categories. Overall the experiments seem lacking at the moment in rigorous comparisons. 
MNIST experimental comparisons are currently suspect. It is  very surprising that LwF does so poorly, do the authors have some explanation for this. LwF is typically a reasonable baseline for these 2 task settings (e.g. https://arxiv.org/pdf/1704.01920.pdf).  Similarly the well known EWC is shown to simply not work at all for the very task it was designed for on the MNIST dataset. LwF and EWC simply not working to any degree seem to me like  rather dramatic claims to make without any explanation. 
Cryptically the fine-tuning baseline described in 4.2 is not shown here for MNIST? This seems a major oversight
CIFAR10/Imagenet Experiments
It is not clear if the baseline finetuning is done on only the top weights or the entire network. Both of these baselines should be considered. Another good baseline to consider is finetuning with cosine distance and only the top weights as in https://arxiv.org/pdf/1804.09458.pdf and other recent works should also be considered
Why do the authors not include any of the baselines from MNIST experiments here, for example LwF.
Ablations study the need for normalization and dynamic margin, it seems these are helpful for accuracy and forward transfer (and not as critical for minimizing forgetting).
The author state their method is agnostic to the task boundaries, its a bit unclear what this means in this context. The procedure is not online and the labels of the samples are being used? If the authors are referring to the need to add additional outputs to the “vanilla” model this seems like it can be trivially addressed by simply saying outputs are added the first time a new class is seen thereby making it agnostic to the boundary in the same sense as this method. 
Clarity 
Can be problematic at times. Although all the elements of the approach are outlined the motivations are overly wordy and repetitive making them actually hard to follow. 
-(minor) first/2nd paragraph of 3.1 seems a bit redundant making it hard to follow
Overall I think the idea to consider metric learning and local adaptation for continual learning is interesting, however the current work is currently lacking in both experimental evidence (appropriate comparisons) and clear motivation/difference to existing work  for its particular instantiation of this idea. 
++++Post Rebuttal++++
Thank you for your detailed responses.
The clarification about “task-agnostic” for the experiments does make them look more relevant than I had previously assessed. I do want to note that the language used for this is inconsistent with the ones used in other papers, which typically calls this a “shared-head” setting (https://arxiv.org/pdf/1801.10112.pdf, https://arxiv.org/pdf/1805.09733.pdf, https://arxiv.org/pdf/1903.08671.pdf ). It is also somewhat inconsistent with the authors own definition of “task agnostic learning” given in the introduction of this paper which implies it is something related to task boundaries at training time, in fact this is something related to availability of the task id at test time. I suggest the authors to make this more clear. Furthermore, the authors should highlight all this in the experiment text, e.g. noting EWC does poorly but this is because we use a different protocol than this and this paper etc.
Regarding the experiments under this light they do look more reasonable. Indeed it has been observed that EWC works poorly in the shared-head setting https://arxiv.org/pdf/1801.10112.pdf
Regarding the new 5 task CIFAR-10 the results are interesting, however I will point the authors to the work above (Rwalk) which also reports results in this setting better than theirs (but not by too much). 
I do however still have issues regarding the memory usage of the method, specifically which data needs to be stored from previous tasks. It is still not completely clear and I find obfuscated since just one sentence not even fully answering the concern about this was added to the manuscript despite myself and another reviewer asking about it. My understanding based on the (somewhat conflicted responses) of the authors is they store a substantial amount of prior task data, but most of this is only used  at test time. For example for imagenet as much as 1000 images/class are stored for testing time. This begs the question why not use this data for training as well if it is allowed to be used by the model at testing time (and therefore preserved from the first task), why is the storage cost of this data not considered and how do the authors justify this still being a lifelong learning setup. As an alternative, why can't one use a much bigger fully parametric model that uses the same amount of storage as the authors model + stored images. It seems it is not fair to compare these to methods that cant utilize this large storage amount. 
Finally its not clear if this data is stored as raw images or somehow stored as embeddings. If it is stored as embeddings this would require some discussion on how the authors avoid representation drift when the next task is training. If the authors store raw images, it means at evaluation time the entire raw dataset needs to be re-encoded, therefore the model can’t perform easily anytime inference. 
Unfortunately the discussion period ended but I would have liked more clarification on this, on the other hand these pieces of information should really have been in the manuscript in the first place. 
Overall, my impression of the paper is improved.  But I do think it could use some further writing revisions to emphasize/clarify key points: a) the method is not new (it says e.g. in abstract “new model” which is misleading) but its application in CL is under-explored b) the experiments show poor performance on existing methods because most of those are not designed nor work well for the shared head “task agnostic” setting, while metric learning handles it gracefully.  c) be explicit about what is the memory being stored when moving onto the next task (this should be somewhere visible and explicit) and how this is justified