Lifelong Learning with Dynamically Expandable Networks
We propose a novel deep network architecture for lifelong learning which we refer to as Dynamically Expandable Network (DEN), that can dynamically decide its network capacity as it trains on a sequence of tasks, to learn a compact overlapping knowledge sharing structure among tasks. DEN is efficiently trained in an online manner by performing selective retraining, dynamically expands network capacity upon arrival of each task with only the necessary number of units, and effectively prevents semantic drift by splitting/duplicating units and timestamping them. We validate DEN on multiple public datasets in lifelong learning scenarios on multiple public datasets, on which it not only significantly outperforms existing lifelong learning methods for deep networks, but also achieves the same level of performance as the batch model with substantially fewer number of parameters.
 In this paper, the authors propose a method (Dynamically Expandable Network) that addresses issues of training efficiency, how to dynamically grow the network, and how to prevent catastrophic forgetting. The paper is well written with a clear problem statement and description of the method for preventing each of the described issues. Interesting points include the use of an L1 regularization term to enforce sparsity in the weights, as well as the method for identifying which neurons have “drifted” too far and should be split. The use of timestamps is a clever addition as well. One question would be how sparse training is done, and how this saves computation, especially with the breadth-first search described on page 5. A critique would be that the base networks (a two layer FF net and LeNet) are not very compelling. Experiments indicate that the method works well, with a clear improvement over progressive networks. Thus, though there isn’t one particular facet of the paper that leaps out, overall the method and results seem solid and worthy of publication.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Lifelong Learning with Dynamically Expandable Networks
We propose a novel deep network architecture for lifelong learning which we refer to as Dynamically Expandable Network (DEN), that can dynamically decide its network capacity as it trains on a sequence of tasks, to learn a compact overlapping knowledge sharing structure among tasks. DEN is efficiently trained in an online manner by performing selective retraining, dynamically expands network capacity upon arrival of each task with only the necessary number of units, and effectively prevents semantic drift by splitting/duplicating units and timestamping them. We validate DEN on multiple public datasets in lifelong learning scenarios on multiple public datasets, on which it not only significantly outperforms existing lifelong learning methods for deep networks, but also achieves the same level of performance as the batch model with substantially fewer number of parameters.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 In this paper, the authors propose a method (Dynamically Expandable Network) that addresses issues of training efficiency, how to dynamically grow the network, and how to prevent catastrophic forgetting. The paper is well written with a clear problem statement and description of the method for preventing each of the described issues. Interesting points include the use of an L1 regularization term to enforce sparsity in the weights, as well as the method for identifying which neurons have “drifted” too far and should be split. The use of timestamps is a clever addition as well. One question would be how sparse training is done, and how this saves computation, especially with the breadth-first search described on page 5. A critique would be that the base networks (a two layer FF net and LeNet) are not very compelling. Experiments indicate that the method works well, with a clear improvement over progressive networks. Thus, though there isn’t one particular facet of the paper that leaps out, overall the method and results seem solid and worthy of publication.
assistant
 In this paper, the authors propose a method (Dynamically Expandable Network) that addresses issues of training efficiency, how to dynamically grow the network, and how to prevent catastrophic forgetting. The paper is well written with a clear problem statement and description of the method for preventing each of the described issues. Interesting points include the use of an L1 regularization term to enforce sparsity in the weights, as well as the method for identifying which neurons have “drifted” too far and should be split. The use of timestamps is a clever addition as well. One question would be how sparse training is done, and how this saves computation, especially with the breadth-first search described on page 5. A critique would be that the base networks (a two layer FF net and LeNet) are not very compelling. Experiments indicate that the method works well, with a clear improvement over progressive networks. Thus, though there isn’t one particular facet of the paper that leaps out, overall the method and results seem solid and worthy of publication.