Continual learning in neural networks (NN) often suffers from catastrophic forgetting. That is, when learning a sequence of tasks on an NN, the learning of a new task will cause weight changes that may destroy the learned knowledge embedded in the weights for previous tasks. Without solving this problem, it is difficult to use an NN to perform continual or lifelong learning. Although researchers have attempted to solve the problem in many ways, it remains to be challenging. In this paper, we propose a new approach, called principal components projection (PCP). The idea is that in learning a new task, if we can ensure that the gradient updates will only occur in the orthogonal directions to the input vectors of the previous tasks, then the weight updates for learning the new task will not affect the previous tasks. We propose to compute the principal components of the input vectors and use them to transform the input and to project the gradient updates for learning each new task. PCP does not need to store any sampled data from previous tasks or to generate pseudo data of previous tasks and use them to help learn a new task. Empirical evaluation shows that the proposed method PCP markedly outperforms the state-of-the-art baseline methods.
The method proposes a method for continual learning. The method is an extension of recent work, called orthogonal weights modification (OWM) [Zheng,2019]. This method aims to find gradient updates which are perpendicular to the input vectors of previous tasks (resulting in less forgetting). However, the authors argue, that the learning of new tasks is happening in the solution space of the previous tasks, which might severely limit the ability to adapt to new tasks. The authors propose a ‘principal component’-based solution to this problem. The method is considering the ‘task continual learning’ scenario (also known as task-aware) which means that the task label is given at inference time.
Conclusion:
1. The paper is not well-positioned in related works. I think the work is more related to works with ‘parameter isolation methods’ such as Piggyback, Packnet, HAT. These methods reserve part of the capacity of the network for tasks. I think the authors should relate their work with these methods, and provide an argument of the problem with these previous methods, which is addressed by their approach. I can see that rather than freezing weights (PackNet) or features (HAT) , the method freezes linear combinations of features. But it is for me not directly clear that that is desirable. In HAT the backpropagated vector is projected on the mask vector which coincides with the neurons (activations). 
2. The experimental verification of the paper is too weak, and only comparison to EWC and OWM (not well known) are provided. At least a comparison with the more related works PackNet and HAT should be included. For more recent method for task-aware CL see also ‘Continual learning: A comparative study on how to defy forgetting in classification tasks’. Also results seem bad. For example on CIFAR10, 5 tasks in TCL setting is two-class problem per task; I would expect better results. 
3. The authors claim that OWM is effective if tasks are similar, but not when dissimilar. And the proposed PCP solves this problem. However, all experiments are on similar tasks, and no cross domain tasks are considered, e.g. going from MNIST (task1) to EMNIST-26 (task2) etc. This would empirically support the claim. Also, the authors expect the difference between PCP and OWM to be even larger then. 
4. Some more analysis of the success of PCA in representing the distribution would be appreciated, e.g. the percentage of total energy which is captured (sum of selected eigenvalues divided by sum of all eigenvalues). Such an analysis of P_l^k as a function of the tasks (and for several layers) would be interesting to see, for example for EMNIST-47(10 tasks). 
5. Novelty with respect to OWM is rather small.
6. The authors should mention that the method is pretrained on ImageNet in section 4.3. Given these datasets, I think it makes more sense to train from scratch and I would like to see those results. 
Minor remarks:
- I wonder if you use OWM or PCP you discard the possibility of positive backward transfer. Maybe the authors could comment on that. 
- The authors write that ‘TCL setting the classification results are usually better than those of the CCL’ is that not per definition true ? Anything correctly classified under CCL is correctly classified under TCL but not the other way around.