Fuzzy paraphrases in learning word representations with a lexicon
A synonym of a polysemous word is usually only the paraphrase of one sense among many. When lexicons are used to improve vector-space word representations, such paraphrases are unreliable and bring noise to the vector-space. The prior works use a coefficient to adjust the overall learning of the lexicons. They regard the paraphrases equally. In this paper, we propose a novel approach that regards the paraphrases diversely to alleviate the adverse effects of polysemy. We annotate each paraphrase with a degree of reliability. The paraphrases are randomly eliminated according to the degrees when our model learns word representations. In this way, our approach drops the unreliable paraphrases, keeping more reliable paraphrases at the same time. The experimental results show that the proposed method improves the word vectors. Our approach is an attempt to address the polysemy problem keeping one vector per word. It makes the approach easier to use than the conventional methods that estimate multiple vectors for a word. Our approach also outperforms the prior works in the experiments.
This paper introduces the concept of fuzzy paraphrases to aid in the learning of distributed word representations from a corpus augmented by a lexicon or ontology. Sometimes polysemy is context-dependent, but prior approaches have neglected this fact when incorporating external paraphrase information during learning. The main idea is to introduce a function that essentially judges the context-sensitivity of paraphrase candidates, down-weighting those candidates that depend strongly on context. This function is inferred from bilingual translation agreement. The main argumentation leading to the model selection is intuitive, and I believe that the inclusion of good paraphrases and the elimination of bad paraphrases during training should in principle improve word representation quality. However, the main questions are how well the proposed method achieves this goal, and, even if it achieves it well, whether it makes much difference in practical terms. Regarding the first question, I am not entirely convinced that the parameterization of the control function f(x_ij) is optimal. It would have been nice to see some experiments investigating different choices, in particular some baselines where the effect of f is diminished (so that it reduces to f=1 in the limit) would have been interesting. I also feel like there would be a lot to gain from having f be a function of the nearby word embeddings, though this would obvious incur a significant slowdown. (See for example *Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space* by Neelakantan et al, which should probably be cited.) As it stands, the experimental results do not clearly distinguish the fuzzy paraphrase approach from prior work, i.e. tables 3 and 4 do not show major trends one way or the other. Regarding the second question, it is hard to draw many conclusions from analogy tasks alone, especially when effects unrelated to good/bad paraphrasing such as corpus size/content, window size, vocabulary size, etc., can have an outsize effect on performance. Overall, I think this is a good paper presenting a sensible idea, but I am not convinced by the experiments that the specific approach is achieving its goal. With some improved experiments and analysis, I would wholeheartedly recommend this paper for acceptance; as it stands, I am on the fence.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Fuzzy paraphrases in learning word representations with a lexicon
A synonym of a polysemous word is usually only the paraphrase of one sense among many. When lexicons are used to improve vector-space word representations, such paraphrases are unreliable and bring noise to the vector-space. The prior works use a coefficient to adjust the overall learning of the lexicons. They regard the paraphrases equally. In this paper, we propose a novel approach that regards the paraphrases diversely to alleviate the adverse effects of polysemy. We annotate each paraphrase with a degree of reliability. The paraphrases are randomly eliminated according to the degrees when our model learns word representations. In this way, our approach drops the unreliable paraphrases, keeping more reliable paraphrases at the same time. The experimental results show that the proposed method improves the word vectors. Our approach is an attempt to address the polysemy problem keeping one vector per word. It makes the approach easier to use than the conventional methods that estimate multiple vectors for a word. Our approach also outperforms the prior works in the experiments.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper introduces the concept of fuzzy paraphrases to aid in the learning of distributed word representations from a corpus augmented by a lexicon or ontology. Sometimes polysemy is context-dependent, but prior approaches have neglected this fact when incorporating external paraphrase information during learning. The main idea is to introduce a function that essentially judges the context-sensitivity of paraphrase candidates, down-weighting those candidates that depend strongly on context. This function is inferred from bilingual translation agreement. The main argumentation leading to the model selection is intuitive, and I believe that the inclusion of good paraphrases and the elimination of bad paraphrases during training should in principle improve word representation quality. However, the main questions are how well the proposed method achieves this goal, and, even if it achieves it well, whether it makes much difference in practical terms. Regarding the first question, I am not entirely convinced that the parameterization of the control function f(x_ij) is optimal. It would have been nice to see some experiments investigating different choices, in particular some baselines where the effect of f is diminished (so that it reduces to f=1 in the limit) would have been interesting. I also feel like there would be a lot to gain from having f be a function of the nearby word embeddings, though this would obvious incur a significant slowdown. (See for example *Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space* by Neelakantan et al, which should probably be cited.) As it stands, the experimental results do not clearly distinguish the fuzzy paraphrase approach from prior work, i.e. tables 3 and 4 do not show major trends one way or the other. Regarding the second question, it is hard to draw many conclusions from analogy tasks alone, especially when effects unrelated to good/bad paraphrasing such as corpus size/content, window size, vocabulary size, etc., can have an outsize effect on performance. Overall, I think this is a good paper presenting a sensible idea, but I am not convinced by the experiments that the specific approach is achieving its goal. With some improved experiments and analysis, I would wholeheartedly recommend this paper for acceptance; as it stands, I am on the fence.
assistant
This paper introduces the concept of fuzzy paraphrases to aid in the learning of distributed word representations from a corpus augmented by a lexicon or ontology. Sometimes polysemy is context-dependent, but prior approaches have neglected this fact when incorporating external paraphrase information during learning. The main idea is to introduce a function that essentially judges the context-sensitivity of paraphrase candidates, down-weighting those candidates that depend strongly on context. This function is inferred from bilingual translation agreement. The main argumentation leading to the model selection is intuitive, and I believe that the inclusion of good paraphrases and the elimination of bad paraphrases during training should in principle improve word representation quality. However, the main questions are how well the proposed method achieves this goal, and, even if it achieves it well, whether it makes much difference in practical terms. Regarding the first question, I am not entirely convinced that the parameterization of the control function f(x_ij) is optimal. It would have been nice to see some experiments investigating different choices, in particular some baselines where the effect of f is diminished (so that it reduces to f=1 in the limit) would have been interesting. I also feel like there would be a lot to gain from having f be a function of the nearby word embeddings, though this would obvious incur a significant slowdown. (See for example *Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space* by Neelakantan et al, which should probably be cited.) As it stands, the experimental results do not clearly distinguish the fuzzy paraphrase approach from prior work, i.e. tables 3 and 4 do not show major trends one way or the other. Regarding the second question, it is hard to draw many conclusions from analogy tasks alone, especially when effects unrelated to good/bad paraphrasing such as corpus size/content, window size, vocabulary size, etc., can have an outsize effect on performance. Overall, I think this is a good paper presenting a sensible idea, but I am not convinced by the experiments that the specific approach is achieving its goal. With some improved experiments and analysis, I would wholeheartedly recommend this paper for acceptance; as it stands, I am on the fence.