Transfer Learning for Sequences via Learning to Collocate
Transfer learning aims to solve the data sparsity for a specific domain by applying information of another domain. Given a sequence (e.g. a natural language sentence), the transfer learning, usually enabled by recurrent neural network (RNN), represent the sequential information transfer. RNN uses a chain of repeating cells to model the sequence data. However, previous studies of neural network based transfer learning simply transfer the information across the whole layers, which are unfeasible for seq2seq and sequence labeling. Meanwhile, such layer-wise transfer learning mechanisms also lose the fine-grained cell-level information from the source domain. In this paper, we proposed the aligned recurrent transfer, ART, to achieve cell-level information transfer. ART is in a recurrent manner that different cells share the same parameters. Besides transferring the corresponding information at the same position, ART transfers information from all collocated words in the source domain. This strategy enables ART to capture the word collocation across domains in a more flexible way. We conducted extensive experiments on both sequence labeling tasks (POS tagging, NER) and sentence classification (sentiment analysis). ART outperforms the state-of-the-arts over all experiments.
 The proposed method is suitable for many NLP tasks, since it can handle the sequence data. I find it difficult to follow through the model descriptions. Perhaps a more descriptive figures would make this easier to follow, I feel that the ART model is a very strait forward and it can be easily described in much simpler and less exhausting (sorry for the strong word) way, while there is nothing wrong with being as elaborating as you are, I feel that all those details belong in an appendix. Can you please explain the exact learning process? I didn’t fully understand the exact way of collocations, you first train on the source domain and then use the trained source network when training in the target domain with all the collocated words for each training example? I deeply encourage you to improve the model section for future readers. In contrast to the model section, the related work and the experimental settings sections are very thin. The experimental setup for the sentiment analysis experiments is quite rare in the transfer learning/domain adaptation landscape, having equal amount of labeled data from both source and target domains is not very realistic in my humble opinion. More realistic setup is unsupervised domain adaptation (like in DANN and MSDA-DAN papers) or minimally supervised domain adaptation (like you did in your POS and NER experiments). In addition to the LSTM baseline (which is trained with target data only), I think that LSTM which is trained on both source and target domains data is required for truly understand ART gains – this goes for the POS and NER tasks as well. The POS and NER experiments can use some additional baselines for further comparison, for example: http://www.aclweb.org/anthology/Q14-1002 https://hornhehhf.github.io/hangfenghe/papers/14484-66685-1-PB.pdf I am not sure I understand the “cell level transfer” claim, did you mean that you are the first to apply inner LSTM/RNN cell transfer or that you are the first ones to apply word-level fine grained transfer, the latter has already been done: https://arxiv.org/pdf/1802.05365.pdf https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=4531&context=sis_research http://www.aclweb.org/anthology/N18-1112 https://openreview.net/pdf?id=rk9eAFcxg
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Transfer Learning for Sequences via Learning to Collocate
Transfer learning aims to solve the data sparsity for a specific domain by applying information of another domain. Given a sequence (e.g. a natural language sentence), the transfer learning, usually enabled by recurrent neural network (RNN), represent the sequential information transfer. RNN uses a chain of repeating cells to model the sequence data. However, previous studies of neural network based transfer learning simply transfer the information across the whole layers, which are unfeasible for seq2seq and sequence labeling. Meanwhile, such layer-wise transfer learning mechanisms also lose the fine-grained cell-level information from the source domain. In this paper, we proposed the aligned recurrent transfer, ART, to achieve cell-level information transfer. ART is in a recurrent manner that different cells share the same parameters. Besides transferring the corresponding information at the same position, ART transfers information from all collocated words in the source domain. This strategy enables ART to capture the word collocation across domains in a more flexible way. We conducted extensive experiments on both sequence labeling tasks (POS tagging, NER) and sentence classification (sentiment analysis). ART outperforms the state-of-the-arts over all experiments.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The proposed method is suitable for many NLP tasks, since it can handle the sequence data. I find it difficult to follow through the model descriptions. Perhaps a more descriptive figures would make this easier to follow, I feel that the ART model is a very strait forward and it can be easily described in much simpler and less exhausting (sorry for the strong word) way, while there is nothing wrong with being as elaborating as you are, I feel that all those details belong in an appendix. Can you please explain the exact learning process? I didn’t fully understand the exact way of collocations, you first train on the source domain and then use the trained source network when training in the target domain with all the collocated words for each training example? I deeply encourage you to improve the model section for future readers. In contrast to the model section, the related work and the experimental settings sections are very thin. The experimental setup for the sentiment analysis experiments is quite rare in the transfer learning/domain adaptation landscape, having equal amount of labeled data from both source and target domains is not very realistic in my humble opinion. More realistic setup is unsupervised domain adaptation (like in DANN and MSDA-DAN papers) or minimally supervised domain adaptation (like you did in your POS and NER experiments). In addition to the LSTM baseline (which is trained with target data only), I think that LSTM which is trained on both source and target domains data is required for truly understand ART gains – this goes for the POS and NER tasks as well. The POS and NER experiments can use some additional baselines for further comparison, for example: http://www.aclweb.org/anthology/Q14-1002 https://hornhehhf.github.io/hangfenghe/papers/14484-66685-1-PB.pdf I am not sure I understand the “cell level transfer” claim, did you mean that you are the first to apply inner LSTM/RNN cell transfer or that you are the first ones to apply word-level fine grained transfer, the latter has already been done: https://arxiv.org/pdf/1802.05365.pdf https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=4531&context=sis_research http://www.aclweb.org/anthology/N18-1112 https://openreview.net/pdf?id=rk9eAFcxg
assistant
 The proposed method is suitable for many NLP tasks, since it can handle the sequence data. I find it difficult to follow through the model descriptions. Perhaps a more descriptive figures would make this easier to follow, I feel that the ART model is a very strait forward and it can be easily described in much simpler and less exhausting (sorry for the strong word) way, while there is nothing wrong with being as elaborating as you are, I feel that all those details belong in an appendix. Can you please explain the exact learning process? I didn’t fully understand the exact way of collocations, you first train on the source domain and then use the trained source network when training in the target domain with all the collocated words for each training example? I deeply encourage you to improve the model section for future readers. In contrast to the model section, the related work and the experimental settings sections are very thin. The experimental setup for the sentiment analysis experiments is quite rare in the transfer learning/domain adaptation landscape, having equal amount of labeled data from both source and target domains is not very realistic in my humble opinion. More realistic setup is unsupervised domain adaptation (like in DANN and MSDA-DAN papers) or minimally supervised domain adaptation (like you did in your POS and NER experiments). In addition to the LSTM baseline (which is trained with target data only), I think that LSTM which is trained on both source and target domains data is required for truly understand ART gains – this goes for the POS and NER tasks as well. The POS and NER experiments can use some additional baselines for further comparison, for example: http://www.aclweb.org/anthology/Q14-1002 https://hornhehhf.github.io/hangfenghe/papers/14484-66685-1-PB.pdf I am not sure I understand the “cell level transfer” claim, did you mean that you are the first to apply inner LSTM/RNN cell transfer or that you are the first ones to apply word-level fine grained transfer, the latter has already been done: https://arxiv.org/pdf/1802.05365.pdf https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=4531&context=sis_research http://www.aclweb.org/anthology/N18-1112 https://openreview.net/pdf?id=rk9eAFcxg