AN EMPIRICAL STUDY OF EXAMPLE FORGETTING DURING DEEP NEURAL NETWORK LEARNING
Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks. Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift. We define a "forgetting event" to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning. Across several benchmark data sets, we find that: (i) certain examples are forgotten with high frequency, and some not at all; (ii) a data set's (un)forgettable examples generalize across neural architectures; and (iii) based on forgetting dynamics, a significant fraction of examples can be omitted from the training data set while still maintaining state-of-the-art generalization performance.
UPDATE 2 (Nov 19, 2018): The paper has improved very substantially since the initial submission, and the authors have addressed almost all of my comments. I have therefore increased my score to an 8 and recommend acceptance.
------------------------------------------------------------------------------------------------------------------------------
UPDATE (Nov 16, 2018) : In light of the author response, I have increased my score to a 6.
------------------------------------------------------------------------------------------------------------------------------
This paper aims to analyze the extent to which networks learn to correctly classify specific examples and then “forget” these examples over the course of training. The authors provide several examples of forgettable and unforgettable examples, demonstrating, among other things, that examples with noisy examples are more forgettable and that a reasonable fraction of unforgettable examples can be removed from the training set without harming performance. 
The paper is clearly written, and the work is novel -- to my knowledge, this is the first investigation of example forgetting over training. There are an interesting and likely important set of ideas here, and portions of the paper are quite strong -- in particular, the experiment demonstrating that examples with noisy examples are more forgettable is quite nice. However, there are several experimental oversights which make this paper difficult to recommend for publication in its current form.
Major points:
1) The most critical issue is with the measurement of forgetting itself: the authors do not take into account the chance forgetting rate in any of their experiments. Simply due to chance, some examples will be correctly labeled at some point in training (especially in the datasets analyzed, which only contain 10 classes). This makes it difficult to distinguish whether a “forgotten” example was actually ever learned in the first place. In order to properly ground this metric, measurements of chance forgetting rates will be necessary (for example, what are the forgetting rates when random steps are taken at each update step?). 
2) Were the networks trained on MNIST, permutedMNIST, and CIFAR-10 trained for the same number of epochs? Related to point 1, the forgetting rate should increase with the number of epochs used in training as the probability of each example being correctly classified should increase. If the CIFAR-10 models were trained for more epochs, this would explain the observation that more CIFAR-10 examples were “forgettable.”
3) In the experiment presented in Figure 4b, it is difficult to tell whether the never forgotten set suffers less degradation in the third training regime because the examples were never forgotten or because the model had twice has much prior experience. Please include a control where the order is flipped (e.g., forgotten, never forgotten, forgotten in addition to the included never forgotten, forgotten, never forgotten order currently present).
4) The visual inspection of forgettable and unforgettable examples in Figure 2 is extremely anecdotal, and moreover, do not even appear to clearly support the claims made in the paper.
Minor points:
1) In the discussion of previous studies which attempted to assess the importance of particular examples to classification decisions, a citation to [1] should be added. 
2) The point regarding similarity across seeds is absolutely critical (especially wrt major comment 1) , and should be included earlier in the paper and more prominently.
3) The histograms in Figure 1 are misleading in the cropped state. While I appreciate that the authors included the full histogram in the supplement, these full histograms should be included in the main figure as well, perhaps as an inset.
4) The inclusion of a space after the commas in numbers (e.g., 50, 245) is quite confusing, especially when multiple numbers are listed as in the first line on page 4.
[1] Koh, Pang Wei and Percy Liang. “Understanding Black-box Predictions via Influence Functions.” ICML (2017).