Scalable and Order-robust Continual Learning with Additive Parameter Decomposition
While recent continual learning methods largely alleviate the catastrophic problem on toy-sized datasets, there are issues that remain to be tackled in order to apply them to real-world problem domains. First, a continual learning model should effectively handle catastrophic forgetting and be efficient to train even with a large number of tasks. Secondly, it needs to tackle the problem of order-sensitivity, where the performance of the tasks largely varies based on the order of the task arrival sequence, as it may cause serious problems where fairness plays a critical role (e.g. medical diagnosis). To tackle these practical challenges, we propose a novel continual learning method that is scalable as well as order-robust, which instead of learning a completely shared set of weights, represents the parameters for each task as a sum of task-shared and sparse task-adaptive parameters. With our Additive Parameter Decomposition (APD), the task-adaptive parameters for earlier tasks remain mostly unaffected, where we update them only to reflect the changes made to the task-shared parameters. This decomposition of parameters effectively prevents catastrophic forgetting and order-sensitivity, while being computation- and memory-efficient. Further, we can achieve even better scalability with APD using hierarchical knowledge consolidation, which clusters the task-adaptive parameters to obtain hierarchically shared parameters. We validate our network with APD, APD-Net, on multiple benchmark datasets against state-of-the-art continual learning methods, which it largely outperforms in accuracy, scalability, and order-robustness.
The paper introduces an additive parameter decomposition (APD) approach to continual learning in the sequential task classification setting and evaluates it across a number of dimensions, including task order robustness, which is comparatively less well researched. Extensive experiments show the novel approach has superior performance to relevant baselines, and provides important data about the order robustness of popular existing approaches. Pros: - Paper tackles the task-order sensitivity challenge in continual learning and introduces an effective order-robust approach. - Method is scalable, parameter growth is logarithmic, forgetting of irrelevant knowledge comes for free. - Baselines are relevant, although they only cover one of the families of approaches. Performance looks consistently better than baselines both in terms of classification accuracy as well as order robustness. - Although non-standard, the Omniglot experiment with 100 tasks is interesting w.r.t. both measures. Performance is on par with STL, also in terms of robustness to order. - Order robustness measures are introduced and motivated. This is particularly relevant for future research beyond simple accuracy comparisons. Cons: - Network architecture used for experiments as well as the exact details of the datasets are non-standard, making results very hard to compare with other papers, so one needs to rely on provided baselines only. Please post citations for papers where the experimental methodologies were adopted from if this is not the case, I many not be familiar with them! - Classification accuracies seems relatively low across the board, especially for CIFAR-100 results. Could you please report some results in the experimental setting used by one of your baselines in the original paper? I am inclined to recommend acceptance due to novelty of order robustness analyses and competitive properties of the method, but I would like clarifications to my experimental questions.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Scalable and Order-robust Continual Learning with Additive Parameter Decomposition
While recent continual learning methods largely alleviate the catastrophic problem on toy-sized datasets, there are issues that remain to be tackled in order to apply them to real-world problem domains. First, a continual learning model should effectively handle catastrophic forgetting and be efficient to train even with a large number of tasks. Secondly, it needs to tackle the problem of order-sensitivity, where the performance of the tasks largely varies based on the order of the task arrival sequence, as it may cause serious problems where fairness plays a critical role (e.g. medical diagnosis). To tackle these practical challenges, we propose a novel continual learning method that is scalable as well as order-robust, which instead of learning a completely shared set of weights, represents the parameters for each task as a sum of task-shared and sparse task-adaptive parameters. With our Additive Parameter Decomposition (APD), the task-adaptive parameters for earlier tasks remain mostly unaffected, where we update them only to reflect the changes made to the task-shared parameters. This decomposition of parameters effectively prevents catastrophic forgetting and order-sensitivity, while being computation- and memory-efficient. Further, we can achieve even better scalability with APD using hierarchical knowledge consolidation, which clusters the task-adaptive parameters to obtain hierarchically shared parameters. We validate our network with APD, APD-Net, on multiple benchmark datasets against state-of-the-art continual learning methods, which it largely outperforms in accuracy, scalability, and order-robustness.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper introduces an additive parameter decomposition (APD) approach to continual learning in the sequential task classification setting and evaluates it across a number of dimensions, including task order robustness, which is comparatively less well researched. Extensive experiments show the novel approach has superior performance to relevant baselines, and provides important data about the order robustness of popular existing approaches. Pros: - Paper tackles the task-order sensitivity challenge in continual learning and introduces an effective order-robust approach. - Method is scalable, parameter growth is logarithmic, forgetting of irrelevant knowledge comes for free. - Baselines are relevant, although they only cover one of the families of approaches. Performance looks consistently better than baselines both in terms of classification accuracy as well as order robustness. - Although non-standard, the Omniglot experiment with 100 tasks is interesting w.r.t. both measures. Performance is on par with STL, also in terms of robustness to order. - Order robustness measures are introduced and motivated. This is particularly relevant for future research beyond simple accuracy comparisons. Cons: - Network architecture used for experiments as well as the exact details of the datasets are non-standard, making results very hard to compare with other papers, so one needs to rely on provided baselines only. Please post citations for papers where the experimental methodologies were adopted from if this is not the case, I many not be familiar with them! - Classification accuracies seems relatively low across the board, especially for CIFAR-100 results. Could you please report some results in the experimental setting used by one of your baselines in the original paper? I am inclined to recommend acceptance due to novelty of order robustness analyses and competitive properties of the method, but I would like clarifications to my experimental questions.
assistant
The paper introduces an additive parameter decomposition (APD) approach to continual learning in the sequential task classification setting and evaluates it across a number of dimensions, including task order robustness, which is comparatively less well researched. Extensive experiments show the novel approach has superior performance to relevant baselines, and provides important data about the order robustness of popular existing approaches. Pros: - Paper tackles the task-order sensitivity challenge in continual learning and introduces an effective order-robust approach. - Method is scalable, parameter growth is logarithmic, forgetting of irrelevant knowledge comes for free. - Baselines are relevant, although they only cover one of the families of approaches. Performance looks consistently better than baselines both in terms of classification accuracy as well as order robustness. - Although non-standard, the Omniglot experiment with 100 tasks is interesting w.r.t. both measures. Performance is on par with STL, also in terms of robustness to order. - Order robustness measures are introduced and motivated. This is particularly relevant for future research beyond simple accuracy comparisons. Cons: - Network architecture used for experiments as well as the exact details of the datasets are non-standard, making results very hard to compare with other papers, so one needs to rely on provided baselines only. Please post citations for papers where the experimental methodologies were adopted from if this is not the case, I many not be familiar with them! - Classification accuracies seems relatively low across the board, especially for CIFAR-100 results. Could you please report some results in the experimental setting used by one of your baselines in the original paper? I am inclined to recommend acceptance due to novelty of order robustness analyses and competitive properties of the method, but I would like clarifications to my experimental questions.