HIERARCHICAL DENSITY ORDER EMBEDDINGS
By representing words with probability densities rather than point vectors, probabilistic word embeddings can capture rich and interpretable semantic information and uncertainty (Vilnis & McCallum, 2014;Athiwaratkun & Wilson, 2017). The uncertainty information can be particularly meaningful in capturing entailment relationships -whereby general words such as "entity" correspond to broad distributions that encompass more specific words such as "animal" or "instrument". We introduce density order embeddings, which learn hierarchical representations through encapsulation of probability distributions. In particular, we propose simple yet effective loss functions and distance metrics, as well as graph-based schemes to select negative samples to better learn hierarchical probabilistic representations. Our approach provides state-of-the-art performance on the WORDNET hypernym relationship prediction task and the challenging HYPERLEX lexical entailment dataset -while retaining a rich and interpretable probabilistic representation.
The paper introduces a novel method for modeling hierarchical data. The work builds on previous approaches, such as Vilnis and McCallum's Word2Gauss and Vendrov's Order Embeddings, to establish a partial order over probability densities via encapsulation, which allows it to model hierarchical information. The aim is to learn embeddings from supervised structured data, such as WordNet. The work also investigates various schemes for selecting negative samples. The evaluation consists of hypernym detection on WordNet and graded lexical entailment, in the shape of HyperLex. This is good work: it is well written, the experiments are thorough and the proposed method is original and works well.
Section 3 could use some more signposting. Especially for 3.3 it would be good to explain (either at the beginning of section 3, or the beginning of section 3.3) why these measures matter and what is going to be done with them.
It's good that LEAR is mentioned and compared against, even though it was very recently published. Please do note that the authors' names are misspelled: Vuli\'c not Vulic, Mrk\v{s}i\'c instead of Mrksic.
If I am not mistaken, the Vendrov WordNet test set is a set of positive pairs. I would like to see more details on how the evaluation is done here: presumably, the lower I set the threshold, the higher my score? Or am I missing something?
It would be useful to describe exactly the extent to which supervision is used - the method only needs positive and negative links, and does not require any additional order information (i.e., WordNet strictly contains more information than what is being used).
I don't see what Socher et al. (2013) has to do with the loss in equation (7). Or did they invent the margin loss?
Word2gauss also evaluates on similarity and relatedness datasets. Did you consider doing that here too?
"hypothesis proposed by Santus et al. which says" is not a valid reference.