A plain well-trained deep learning model often does not have the ability to learn new knowledge without forgetting the previously learned knowledge, which is known as catastrophic forgetting. Here we propose a novel method, SupportNet, to efficiently and effectively solve the catastrophic forgetting problem in the class incremental learning scenario. SupportNet combines the strength of deep learning and support vector machine (SVM), where SVM is used to identify the support data from the old data, which are fed to the deep learning model together with the new data for further training so that the model can review the essential information of the old data when learning the new information. Two powerful consolidation regularizers are applied to stabilize the learned representation and ensure the robustness of the learned model. We validate our method with comprehensive experiments on various tasks, which show that SupportNet drastically outperforms the state-of-the-art incremental learning methods and even reaches similar performance as the deep learning model trained from scratch on both old and new data.
Summary:
The authors offer a novel incremental learning method called SupportNet to combat catastrophic forgetting that can be seen in standard deep learning models. Catastrophic forgetting is the phenomenon where the networks don’t retain old knowledge when they learn new knowledge.  SupportNet uses resnet network with 32 layers, trains an SVM on the last layer and the support vector points from this SVM are given to the network along with the new data. Furthermore, two regularizers, feature and EWC regularizer, are added to the network. The feature regularizer forces the network to produce fixed representation for the old data, since if the feature representation for the old data changes when the network is fine-tuned on the new data then the support vectors generated from the old feature representation of the old data would become invalid.  The EWC regularizer works by constraining parameters crucial for the classification of the old data, making it harder for the network to change them. SupportNet is compared to five methods (all data: network is re-trained with new and old data, upper bound for performance, iCaRL: state-of-the-art method for incremental learning, EWC: Only EWC regularizer added, Fine tune: Only new data, Random guessing: Random guess to assign labels) on six datasets (MNIST, CIFAR-10, CIFAR-100, Enzyme Function Prediction, HeLa Subcellular Structure Classification, Breast Tumor Classification). It shows some improvement in overall accuracy with each newly added class when compared to iCaRL, EWC, Fine Tune and Random guessing.  Additionally, they show that overfitting for the real training data (a chosen subset of old data and the new data) is a problem for the competition iCaRL and affects SupportNet to a much lesser degree.
Pros:
(1)
The authors propose a sensible approach, which is also novel to be best of our knowledge, using SVM to select support data from old data to be fed to the network along with the new data in the incremental learning framework to avoid catastrophic forgetting. Additionally, they offer a feature regularizer that penalizes the network for changing the feature representation of the support data when training the network on new data and an EWC regularizer that constrains the parameters that are crucial for the classification of the old data and makes it harder to change them. 
(2)
The authors use six different datasets and several other approaches (subsets of their method’s components, other competing methods) to show these three components alleviate catastrophic forgetting and show improvement in overall accuracy.
(3)
The paper is well written and easy to follow. 
Cons:
Major Points:
(1)
To show that the method proposed in the paper addresses catastrophic forgetting, in addition to the overall accuracy shown in Figure 3, it is also necessary to show the accuracy of different models on old classes when new classes are added to the network. This will strengthen the argument that the improvement in accuracy is indeed due to correct classification on old data.
 (2)
The authors claim that iCaRL suffers from overfitting on real training data (section 4.1) however Table 2 shows iCaRL only on the enzyme function prediction which is also the dataset where the difference in performance between iCaRL and SupportNet is the largest. To support the general overfitting claim made in section 4.1, the authors should repeat this analysis on any of the other five datasets where the performance difference between the two methods is much smaller.  SupportNet also suffers from overfitting (Table 3, Accuracy: test data: 83.9%, real training data: 98.7%) although to a lesser extent than iCaRL.
(3)
The individual impact of the support points and the joint impact of support points with feature regularizer on accuracy is not assessed. To prove their usefulness, add two methods to Figure 3:
(a)A method that uses support points without any regularizer.
(b) A method that uses support points with just the feature regularizer. 
Other points:
(1) 
In section 2.3.2, EWC regularizer, Eq. 9: We think F(theta_new) should be F(theta_old) since we want to constrain parameters crucial for classification of old data and should be computing Fisher Information for the old parameters. 
(2)
In section 2.1 Deep Learning and SVM: additional steps are needed to show how Eq. 3 is derived from Eq. 2.
(3)
In section 2.1 Deep Learning and SVM: In the line before Eq. 4. “t represtent” instead of “t represents”.
(4)
Figures are small and hard to read. Please increase the size and resolution of the figures.