Representation Stability as a Regularizer for Improved Text Analytics Transfer Learning
Although neural networks are well suited for sequential transfer learning tasks, the catastrophic forgetting problem hinders proper integration of prior knowledge. In this work, we propose a solution to this problem by using a multi-task objective based on the idea of distillation and a mechanism that directly penalizes forgetting at the shared representation layer during the knowledge integration phase of training. We demonstrate our approach on a Twitter domain sentiment analysis task with sequential knowledge transfer from four related tasks. We show that our technique outperforms networks fine-tuned to the target task. Additionally, we show both through empirical evidence and examples that it does not forget useful knowledge from the source task that is forgotten during standard fine-tuning. Surprisingly, we find that first distilling a human made rule based sentiment engine into a recurrent neural network and then integrating the knowledge with the target task data leads to a substantial gain in generalization performance. Our experiments demonstrate the power of multi-source transfer techniques in practical text analytics problems when paired with distillation. In particular, for the SemEval 2016 Task 4 Subtask A (Nakov et al., 2016) dataset we surpass the state of the art established during the competition with a comparatively simple model architecture that is not even competitive when trained on only the labeled task specific data.
This paper proposes a regularization technique for neural network training that relies on having multiple related tasks or datasets in a transfer learning setting. The proposed technique is straightforward to describe and can also leverage external labeling systems perhaps based on logical rules. The paper is clearly written and the experiments seem relatively thorough. Overall this is a nice paper but does not fully address how robust the proposed technique is. For each experiment there seems to be a slightly different application of the proposed technique, or a lot of ensembling and cross validation. I can’t figure out if this is because the proposed technique does not work well in general and thus required a lot of fiddling to get right in experiments, or if this is simply an artifact of ad-hoc experiments to try and get the best performance overall. If more datasets or addressing this issue directly in discussion was able to show this the strengths and limitations of the proposed technique more clearly, this could be a great paper. Overall the proposed method seems nice and possibly useful for other problems. However in the details of logical rule distillation and various experiment settings it seems like there is a lot of running the model many times or selecting a particular way of reusing the models and data that makes me wonder how robust the technique is or whether it requires a lot of trying various approaches, ensembling, or picking the best model from cross validation to show real gains. The authors could help by discussing this explicitly for all experiments in one place rather than listing the various choices / approaches in each experiment. As an example, these sorts of phrases make me very unsure how reliable the method is in practice versus how much the authors had to engineer this regularizer to perform well: “We noticed that equation 8 is actually prone to overfitting away from a good solution on the test set although it often finds a pretty good one early in training. “ The introduction section should first review the definitions of transfer learning vs multi-task learning to make the discussion more clear. It also deems justification why “catastrophic forgetting” is actually a problem. If the final target task is the only thing of interest then forgetting the source task is not an issue and the authors should motivate why forgetting matters in their setting. This paper explores sequential transfer so it’s not obvious why forgetting the source task matters. Section 7 introduces the logical rules engine in a fairly specific context. Rather it would be good state more generally what this system entails to help people figure out how this method would apply to other problems.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Representation Stability as a Regularizer for Improved Text Analytics Transfer Learning
Although neural networks are well suited for sequential transfer learning tasks, the catastrophic forgetting problem hinders proper integration of prior knowledge. In this work, we propose a solution to this problem by using a multi-task objective based on the idea of distillation and a mechanism that directly penalizes forgetting at the shared representation layer during the knowledge integration phase of training. We demonstrate our approach on a Twitter domain sentiment analysis task with sequential knowledge transfer from four related tasks. We show that our technique outperforms networks fine-tuned to the target task. Additionally, we show both through empirical evidence and examples that it does not forget useful knowledge from the source task that is forgotten during standard fine-tuning. Surprisingly, we find that first distilling a human made rule based sentiment engine into a recurrent neural network and then integrating the knowledge with the target task data leads to a substantial gain in generalization performance. Our experiments demonstrate the power of multi-source transfer techniques in practical text analytics problems when paired with distillation. In particular, for the SemEval 2016 Task 4 Subtask A (Nakov et al., 2016) dataset we surpass the state of the art established during the competition with a comparatively simple model architecture that is not even competitive when trained on only the labeled task specific data.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper proposes a regularization technique for neural network training that relies on having multiple related tasks or datasets in a transfer learning setting. The proposed technique is straightforward to describe and can also leverage external labeling systems perhaps based on logical rules. The paper is clearly written and the experiments seem relatively thorough. Overall this is a nice paper but does not fully address how robust the proposed technique is. For each experiment there seems to be a slightly different application of the proposed technique, or a lot of ensembling and cross validation. I can’t figure out if this is because the proposed technique does not work well in general and thus required a lot of fiddling to get right in experiments, or if this is simply an artifact of ad-hoc experiments to try and get the best performance overall. If more datasets or addressing this issue directly in discussion was able to show this the strengths and limitations of the proposed technique more clearly, this could be a great paper. Overall the proposed method seems nice and possibly useful for other problems. However in the details of logical rule distillation and various experiment settings it seems like there is a lot of running the model many times or selecting a particular way of reusing the models and data that makes me wonder how robust the technique is or whether it requires a lot of trying various approaches, ensembling, or picking the best model from cross validation to show real gains. The authors could help by discussing this explicitly for all experiments in one place rather than listing the various choices / approaches in each experiment. As an example, these sorts of phrases make me very unsure how reliable the method is in practice versus how much the authors had to engineer this regularizer to perform well: “We noticed that equation 8 is actually prone to overfitting away from a good solution on the test set although it often finds a pretty good one early in training. “ The introduction section should first review the definitions of transfer learning vs multi-task learning to make the discussion more clear. It also deems justification why “catastrophic forgetting” is actually a problem. If the final target task is the only thing of interest then forgetting the source task is not an issue and the authors should motivate why forgetting matters in their setting. This paper explores sequential transfer so it’s not obvious why forgetting the source task matters. Section 7 introduces the logical rules engine in a fairly specific context. Rather it would be good state more generally what this system entails to help people figure out how this method would apply to other problems.
assistant
This paper proposes a regularization technique for neural network training that relies on having multiple related tasks or datasets in a transfer learning setting. The proposed technique is straightforward to describe and can also leverage external labeling systems perhaps based on logical rules. The paper is clearly written and the experiments seem relatively thorough. Overall this is a nice paper but does not fully address how robust the proposed technique is. For each experiment there seems to be a slightly different application of the proposed technique, or a lot of ensembling and cross validation. I can’t figure out if this is because the proposed technique does not work well in general and thus required a lot of fiddling to get right in experiments, or if this is simply an artifact of ad-hoc experiments to try and get the best performance overall. If more datasets or addressing this issue directly in discussion was able to show this the strengths and limitations of the proposed technique more clearly, this could be a great paper. Overall the proposed method seems nice and possibly useful for other problems. However in the details of logical rule distillation and various experiment settings it seems like there is a lot of running the model many times or selecting a particular way of reusing the models and data that makes me wonder how robust the technique is or whether it requires a lot of trying various approaches, ensembling, or picking the best model from cross validation to show real gains. The authors could help by discussing this explicitly for all experiments in one place rather than listing the various choices / approaches in each experiment. As an example, these sorts of phrases make me very unsure how reliable the method is in practice versus how much the authors had to engineer this regularizer to perform well: “We noticed that equation 8 is actually prone to overfitting away from a good solution on the test set although it often finds a pretty good one early in training. “ The introduction section should first review the definitions of transfer learning vs multi-task learning to make the discussion more clear. It also deems justification why “catastrophic forgetting” is actually a problem. If the final target task is the only thing of interest then forgetting the source task is not an issue and the authors should motivate why forgetting matters in their setting. This paper explores sequential transfer so it’s not obvious why forgetting the source task matters. Section 7 introduces the logical rules engine in a fairly specific context. Rather it would be good state more generally what this system entails to help people figure out how this method would apply to other problems.