Lifelong Word Embedding via Meta-Learning
Learning high-quality word embeddings is of significant importance in achieving better performance in many down-stream learning tasks. On one hand, traditional word embeddings are trained on a large scale corpus for general-purpose tasks, which are often sub-optimal for many domain-specific tasks. On the other hand, many domain-specific tasks do not have a large enough domain corpus to obtain high-quality embeddings. We observe that domains are not isolated and a small domain corpus can leverage the learned knowledge from many past domains to augment that corpus in order to generate high-quality embeddings. In this paper, we formulate the learning of word embeddings as a lifelong learning process. Given knowledge learned from many previous domains and a small new domain corpus, the proposed method can effectively generate new domain embeddings by leveraging a simple but effective algorithm and a meta-learner, where the meta-learner is able to provide word context similarity information at the domain-level. Experimental results demonstrate that the proposed method can effectively learn new domain embeddings from a small corpus and past domain knowledgesfootnote{We will release the code after final revisions.}. We also demonstrate that general-purpose embeddings trained from a large scale corpus are sub-optimal in domain-specific tasks.
 In this paper, the authors proposed to learn word embedding for the target domain in the lifelong learning manner. The basic idea is to learn a so-call meter learner to measure similarities of the same words between the target domain and the source domains for help learning word embedding for the target domain with a small corpus. Overall, the descriptions of the proposed model (Section 3 - Section 5) are hard to follow. This is not because the proposed model is technically difficult to understand. On the contrary, the model is heuristic, and simple, but the descriptions are unclear. Section 3 is supposed to give an overview and high-level introduction of the whole model using the Figure 1, and Figure 2 (not Figure 3 mentioned in text). However, after reading Section 3, I do not catch any useful information about the proposed model expect for knowing that a so-called meta learner is used. Section 4 and Section 5 are supposed to give details of different components of the proposed model and explain the motivations. However, descriptions in these two sections are very confusing, e.g, many symbols in Algorithm 1 are presented with any descriptions. Moreover, the motivations behind the proposed methods for different components are missing. Also, a lot of types make the descriptions more difficult to follow, e.g., "may not helpful or even harmful", *"Figure 3", "we show this Section 6", "large size a vocabulary", etc. Another major concern is that the technical contributions of the proposed model is quite limited. The only technical contributions are (4) and the way to construct the co-occurrence information A. However, such contributions are quite minor, and technically heuristic. Moreover, regarding the aggregation layer in the pairwise network, it is similar to feature engineering. In this case, why not just train a flat classifier, like logistic regression, with rich feature engineering, in stead of using a neural network. Regarding experiments, one straight-forward baseline is missing. As n domains are supposed to be given in advance before the n+1 domain (target domain) comes, one can use multi-domain learning approaches with ensemble learning techniques to learn word embedding for the target domain. For instance, one can learn n pairwise (1 out of n sources + the target) cross-domain word embedding, and combine them using the similarity between each source and the target as the weight.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Lifelong Word Embedding via Meta-Learning
Learning high-quality word embeddings is of significant importance in achieving better performance in many down-stream learning tasks. On one hand, traditional word embeddings are trained on a large scale corpus for general-purpose tasks, which are often sub-optimal for many domain-specific tasks. On the other hand, many domain-specific tasks do not have a large enough domain corpus to obtain high-quality embeddings. We observe that domains are not isolated and a small domain corpus can leverage the learned knowledge from many past domains to augment that corpus in order to generate high-quality embeddings. In this paper, we formulate the learning of word embeddings as a lifelong learning process. Given knowledge learned from many previous domains and a small new domain corpus, the proposed method can effectively generate new domain embeddings by leveraging a simple but effective algorithm and a meta-learner, where the meta-learner is able to provide word context similarity information at the domain-level. Experimental results demonstrate that the proposed method can effectively learn new domain embeddings from a small corpus and past domain knowledgesfootnote{We will release the code after final revisions.}. We also demonstrate that general-purpose embeddings trained from a large scale corpus are sub-optimal in domain-specific tasks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 In this paper, the authors proposed to learn word embedding for the target domain in the lifelong learning manner. The basic idea is to learn a so-call meter learner to measure similarities of the same words between the target domain and the source domains for help learning word embedding for the target domain with a small corpus. Overall, the descriptions of the proposed model (Section 3 - Section 5) are hard to follow. This is not because the proposed model is technically difficult to understand. On the contrary, the model is heuristic, and simple, but the descriptions are unclear. Section 3 is supposed to give an overview and high-level introduction of the whole model using the Figure 1, and Figure 2 (not Figure 3 mentioned in text). However, after reading Section 3, I do not catch any useful information about the proposed model expect for knowing that a so-called meta learner is used. Section 4 and Section 5 are supposed to give details of different components of the proposed model and explain the motivations. However, descriptions in these two sections are very confusing, e.g, many symbols in Algorithm 1 are presented with any descriptions. Moreover, the motivations behind the proposed methods for different components are missing. Also, a lot of types make the descriptions more difficult to follow, e.g., "may not helpful or even harmful", *"Figure 3", "we show this Section 6", "large size a vocabulary", etc. Another major concern is that the technical contributions of the proposed model is quite limited. The only technical contributions are (4) and the way to construct the co-occurrence information A. However, such contributions are quite minor, and technically heuristic. Moreover, regarding the aggregation layer in the pairwise network, it is similar to feature engineering. In this case, why not just train a flat classifier, like logistic regression, with rich feature engineering, in stead of using a neural network. Regarding experiments, one straight-forward baseline is missing. As n domains are supposed to be given in advance before the n+1 domain (target domain) comes, one can use multi-domain learning approaches with ensemble learning techniques to learn word embedding for the target domain. For instance, one can learn n pairwise (1 out of n sources + the target) cross-domain word embedding, and combine them using the similarity between each source and the target as the weight.
assistant
 In this paper, the authors proposed to learn word embedding for the target domain in the lifelong learning manner. The basic idea is to learn a so-call meter learner to measure similarities of the same words between the target domain and the source domains for help learning word embedding for the target domain with a small corpus. Overall, the descriptions of the proposed model (Section 3 - Section 5) are hard to follow. This is not because the proposed model is technically difficult to understand. On the contrary, the model is heuristic, and simple, but the descriptions are unclear. Section 3 is supposed to give an overview and high-level introduction of the whole model using the Figure 1, and Figure 2 (not Figure 3 mentioned in text). However, after reading Section 3, I do not catch any useful information about the proposed model expect for knowing that a so-called meta learner is used. Section 4 and Section 5 are supposed to give details of different components of the proposed model and explain the motivations. However, descriptions in these two sections are very confusing, e.g, many symbols in Algorithm 1 are presented with any descriptions. Moreover, the motivations behind the proposed methods for different components are missing. Also, a lot of types make the descriptions more difficult to follow, e.g., "may not helpful or even harmful", *"Figure 3", "we show this Section 6", "large size a vocabulary", etc. Another major concern is that the technical contributions of the proposed model is quite limited. The only technical contributions are (4) and the way to construct the co-occurrence information A. However, such contributions are quite minor, and technically heuristic. Moreover, regarding the aggregation layer in the pairwise network, it is similar to feature engineering. In this case, why not just train a flat classifier, like logistic regression, with rich feature engineering, in stead of using a neural network. Regarding experiments, one straight-forward baseline is missing. As n domains are supposed to be given in advance before the n+1 domain (target domain) comes, one can use multi-domain learning approaches with ensemble learning techniques to learn word embedding for the target domain. For instance, one can learn n pairwise (1 out of n sources + the target) cross-domain word embedding, and combine them using the similarity between each source and the target as the weight.