A COMPREHENSIVE, APPLICATION-ORIENTED STUDY OF CATASTROPHIC FORGETTING IN DNNS
We present a large-scale empirical study of catastrophic forgetting (CF) in modern Deep Neural Network (DNN) models that perform sequential (or: incremental) learning. A new experimental protocol is proposed that enforces typical constraints encountered in application scenarios. As the investigation is empirical, we evaluate CF behavior on the hitherto largest number of visual classification datasets, from each of which we construct a representative number of Sequential Learning Tasks (SLTs) in close alignment to previous works on CF. Our results clearly indicate that there is no model that avoids CF for all investigated datasets and SLTs under application conditions. We conclude with a discussion of potential solutions and workarounds to CF, notably for the EWC and IMM models.
# [Updated after author response]
Thank you for your response. I am happy to see the updated paper. In particular, the added item in section 1.3 highlights where the novelty of the paper lies, and as a consequence, I think the significance of the paper is increased. Furthermore, the clarity of the paper has increased. 
In its current form, I think the paper would be a valuable input to the deep learning community, highlighting an important issue (CF) for neural networks. I have therefore increased my score.
------------------------------------------
# Summary
The authors present an empirical study of catastrophic forgetting (CF) in deep neural networks. Eight models are tested against nine datasets with 10 classes each but a varying number of samples. The authors construct a number of sequential learning tasks to test the model performances in different scenarios. The main conclusion is that CF is still a problem in all models, despite claims in other papers.
# Quality
The paper shows healthy criticism of the methods used to evaluate CF in previous works. I very much like this.
While I like the different experimental set-ups and the attention to realistic scenarios outlined in section 1.2, I find the analysis of the experiments somewhat superficial. The accuracies of each model for each task and dataset are reported, but there is little insight into what causes CF. For instance, do some choices of hyperparameters consistently cause a higher/lower degree of CF across models? I also think the metrics proposed by Kemker et al. (2018) are more informative than just reporting the last and best accuracy, and that including these metrics would improve the quality of the paper.
# Clarity
The paper is generally clearly written and distinct paragraphs are often highlighted, which makes reading and getting an overview much easier. In particular, I like the summary given in sections 1.3 and 1.4.
Section 2.4 describing the experimental setup could be clearer. It takes a bit of time to decipher Table 2, and it would have been good with a few short comments on what the different types of tasks (D5-5, D9-1, DP10-10) will tell us about the model performances. E.g. what do you expect to see from the experiments of D5-5 that is not covered by D9-1 and vice versa? And why are the number of tasks in each category so different (8 vs 3 vs 1)?
I am not a huge fan of 3D plots, and I don't think they do anything good in section 4. The perspective can make it tricky to compare models, and the different graphs overshadow each other. I would prefer 2D plots in the supplementary, with a few representative ones shown in the main paper. I would also experiment with turning Table 3 into a heat map.
# Originality
To my knowledge, the paper presents the largest evaluation of CF in terms of evaluated datasets. Kemker et al. (2018) conduct a somewhat similar experiment using fewer datasets, but a larger number of classes, which makes the CF even clearer. I think it would be good to cite this paper and briefly discuss it in connection with the current work.
# Significance
The paper is mostly a report of the outcome of a substantial experiment on CF, showing that all tested models suffer from CF to some extent. While this is interesting and useful to know, there is not much to learn in terms of what can cause or prevent CF in DNNs. The paper's significance lies in showing that CF is still a problem, but there is room for improvement in the analysis of the outcome of the experiments.
# Other notes
The first sentence of the second paragraph in section 5 seems to be missing something.
# References
Kemker, R., McClure, M., Abitino, A., Hayes, T., & Kanan, C. (2018). In AAAI Conference on Artificial Intelligence. https://aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16410