Distributional Inclusion Vector Embedding for Unsupervised Hypernymy Detection
Modeling hypernymy, such as poodle is-a dog, is an important generalization aid to many NLP tasks, such as entailment, relation extraction, and question answering. Supervised learning from labeled hypernym sources, such as WordNet, limit the coverage of these models, which can be addressed by learning hypernyms from unlabeled text. Existing unsupervised methods either do not scale to large vocabularies or yield unacceptably poor accuracy. This paper introduces {it distributional inclusion vector embedding (DIVE)}, a simple-to-implement unsupervised method of hypernym discovery via per-word non-negative vector embeddings which preserve the inclusion property of word contexts. In experimental evaluations more comprehensive than any previous literature of which we are aware---evaluating on 11 datasets using multiple existing as well as newly proposed scoring functions---we find that our method provides up to double the precision of previous unsupervised methods, and the highest average performance, using a much more compact word representation, and yielding many new state-of-the-art results. In addition, the meaning of each dimension in DIVE is interpretable, which leads to a novel approach on word sense disambiguation as another promising application of DIVE.
 This paper presents a new method for detecting hypernymy by extending the distributional inclusion hypothesis to low-rank embeddings. The first half of the paper is written superbly, providing a sober account of the state-of-the-art in hypernymy detection via distributional semantics, and a clear, well-motivated explanation of the main algorithm (DIVE). The reformulation of PMI is interesting; the authors essentially replace P(w) with the uniform distribution (this should be stated explicitly in the paper, BTW). They then augment SGNS to reflect this change by dynamically selecting the number of negatively-sampled contexts according to the target word (w). This is a clever trick. I was also very happy to see that the authors evaluated on virtually every lexical inference dataset available. However, the remainder of the paper describing the experiments and their results was extremely difficult to parse. First of all, the line between the authors* contribution and prior art is blurry, because they seem to be introducing new metrics for measuring hypernymy as part of the experiments. There are also too many moving parts: datasets, evaluation metrics, detection metrics, embedding methods, hyperparameters, etc. These need to be organized, controlled for, and clearly explained. I have two major concerns regarding the experiments, beyond intelligibility. First, the authors used a specific corpus (with specific preprocessing) to train their vectors, but compared their results to those reported in other papers, which are not based on the same data. This invalidates these comparisons. Instead, the other methods should be replicated and rerun on exactly the same corpus. I would also recommend using a much larger corpus, since 50M tokens is considered quite small when training word embeddings. My second concern is that summation (dS) is doing all the heavy lifting. In Table 2, we can see that the difference is only 0.1 between dS and W * dS (where W is also not trained using DIVE). Since dS is basically a proxy for difference in word frequency, could it be that the proposed method is just computing which word is more general? This looks awfully familiar to Levy et al*s prototypical hypernym result. Miscellaneous Comments: - There*s a very recent paper on using vector norms for detecting hypernymy that might be worth contrasting with: https://arxiv.org/pdf/1710.06371.pdf - Micro-averaging these datasets is problematic, because some of the datasets based on WordNet are much larger than the hand-annotated ones, and will likely drown them out. Because these datasets are so different, I think it is critical to look at the details and not only at the averages. - PMI filtering needs to be controlled/ablated in the experiments. - While explaining equation 6, the authors say that the gradients for x and y are similar; this is not true, because k is a function of x (or y), and if one appears more often in general (not necessarily with c), then the gradients will be different as well.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Distributional Inclusion Vector Embedding for Unsupervised Hypernymy Detection
Modeling hypernymy, such as poodle is-a dog, is an important generalization aid to many NLP tasks, such as entailment, relation extraction, and question answering. Supervised learning from labeled hypernym sources, such as WordNet, limit the coverage of these models, which can be addressed by learning hypernyms from unlabeled text. Existing unsupervised methods either do not scale to large vocabularies or yield unacceptably poor accuracy. This paper introduces {it distributional inclusion vector embedding (DIVE)}, a simple-to-implement unsupervised method of hypernym discovery via per-word non-negative vector embeddings which preserve the inclusion property of word contexts. In experimental evaluations more comprehensive than any previous literature of which we are aware---evaluating on 11 datasets using multiple existing as well as newly proposed scoring functions---we find that our method provides up to double the precision of previous unsupervised methods, and the highest average performance, using a much more compact word representation, and yielding many new state-of-the-art results. In addition, the meaning of each dimension in DIVE is interpretable, which leads to a novel approach on word sense disambiguation as another promising application of DIVE.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper presents a new method for detecting hypernymy by extending the distributional inclusion hypothesis to low-rank embeddings. The first half of the paper is written superbly, providing a sober account of the state-of-the-art in hypernymy detection via distributional semantics, and a clear, well-motivated explanation of the main algorithm (DIVE). The reformulation of PMI is interesting; the authors essentially replace P(w) with the uniform distribution (this should be stated explicitly in the paper, BTW). They then augment SGNS to reflect this change by dynamically selecting the number of negatively-sampled contexts according to the target word (w). This is a clever trick. I was also very happy to see that the authors evaluated on virtually every lexical inference dataset available. However, the remainder of the paper describing the experiments and their results was extremely difficult to parse. First of all, the line between the authors* contribution and prior art is blurry, because they seem to be introducing new metrics for measuring hypernymy as part of the experiments. There are also too many moving parts: datasets, evaluation metrics, detection metrics, embedding methods, hyperparameters, etc. These need to be organized, controlled for, and clearly explained. I have two major concerns regarding the experiments, beyond intelligibility. First, the authors used a specific corpus (with specific preprocessing) to train their vectors, but compared their results to those reported in other papers, which are not based on the same data. This invalidates these comparisons. Instead, the other methods should be replicated and rerun on exactly the same corpus. I would also recommend using a much larger corpus, since 50M tokens is considered quite small when training word embeddings. My second concern is that summation (dS) is doing all the heavy lifting. In Table 2, we can see that the difference is only 0.1 between dS and W * dS (where W is also not trained using DIVE). Since dS is basically a proxy for difference in word frequency, could it be that the proposed method is just computing which word is more general? This looks awfully familiar to Levy et al*s prototypical hypernym result. Miscellaneous Comments: - There*s a very recent paper on using vector norms for detecting hypernymy that might be worth contrasting with: https://arxiv.org/pdf/1710.06371.pdf - Micro-averaging these datasets is problematic, because some of the datasets based on WordNet are much larger than the hand-annotated ones, and will likely drown them out. Because these datasets are so different, I think it is critical to look at the details and not only at the averages. - PMI filtering needs to be controlled/ablated in the experiments. - While explaining equation 6, the authors say that the gradients for x and y are similar; this is not true, because k is a function of x (or y), and if one appears more often in general (not necessarily with c), then the gradients will be different as well.
assistant
 This paper presents a new method for detecting hypernymy by extending the distributional inclusion hypothesis to low-rank embeddings. The first half of the paper is written superbly, providing a sober account of the state-of-the-art in hypernymy detection via distributional semantics, and a clear, well-motivated explanation of the main algorithm (DIVE). The reformulation of PMI is interesting; the authors essentially replace P(w) with the uniform distribution (this should be stated explicitly in the paper, BTW). They then augment SGNS to reflect this change by dynamically selecting the number of negatively-sampled contexts according to the target word (w). This is a clever trick. I was also very happy to see that the authors evaluated on virtually every lexical inference dataset available. However, the remainder of the paper describing the experiments and their results was extremely difficult to parse. First of all, the line between the authors* contribution and prior art is blurry, because they seem to be introducing new metrics for measuring hypernymy as part of the experiments. There are also too many moving parts: datasets, evaluation metrics, detection metrics, embedding methods, hyperparameters, etc. These need to be organized, controlled for, and clearly explained. I have two major concerns regarding the experiments, beyond intelligibility. First, the authors used a specific corpus (with specific preprocessing) to train their vectors, but compared their results to those reported in other papers, which are not based on the same data. This invalidates these comparisons. Instead, the other methods should be replicated and rerun on exactly the same corpus. I would also recommend using a much larger corpus, since 50M tokens is considered quite small when training word embeddings. My second concern is that summation (dS) is doing all the heavy lifting. In Table 2, we can see that the difference is only 0.1 between dS and W * dS (where W is also not trained using DIVE). Since dS is basically a proxy for difference in word frequency, could it be that the proposed method is just computing which word is more general? This looks awfully familiar to Levy et al*s prototypical hypernym result. Miscellaneous Comments: - There*s a very recent paper on using vector norms for detecting hypernymy that might be worth contrasting with: https://arxiv.org/pdf/1710.06371.pdf - Micro-averaging these datasets is problematic, because some of the datasets based on WordNet are much larger than the hand-annotated ones, and will likely drown them out. Because these datasets are so different, I think it is critical to look at the details and not only at the averages. - PMI filtering needs to be controlled/ablated in the experiments. - While explaining equation 6, the authors say that the gradients for x and y are similar; this is not true, because k is a function of x (or y), and if one appears more often in general (not necessarily with c), then the gradients will be different as well.