Entailment vectors are a principled way to encode in a vector what information is known and what is unknown. They are designed to model relations where one vector should include all the information in another vector, called entailment. This paper investigates the unsupervised learning of entailment vectors for the semantics of words. Using simple entailment-based models of the semantics of words in text (distributional semantics), we induce entailment-vector word embeddings which outperform the best previous results for predicting entailment between words, in unsupervised and semi-supervised experiments on hyponymy.
This work proposes to learn word vectors that are intended to specifically model the lexical entailment relationship. This is achieved in an unsupervised manner from unstructured data, through an approach heavily influenced by recent work by Henderson and Popa, which "reinterprets word2vec" by modeling distributions over discrete latent "pseudo-phrase" vectors. That is, instead of using two vectors per word, as in word2vec, a latent representation is introduced that models the joint properties of the target and context words. While Henderson and Popa represent the latent vector as the evidence for the target and context, or the likelihood, this work suggests to represent it based on the posterior distribution instead. The resultant representations are evaluated on Weeds et al.'s (2014) version of BLESS, as well as the full BLESS dataset, where they do better than the original.
The paper is confusingly written, fails to mention a lot of related work, has a weak evaluation where it doesn't compare to related systems, and I feel that it would benefit from "toning down". Hence, I do not recommend it for acceptance. In more detail:
1. The idea behind Henderson and Popa's model, as well as the suggested modification, should be easy to explain, but I really had to struggle to make sense of it. This work relies very heavily on that paper, and would be better off if it was more standalone. I think part of the confusion stems from using y for the latent representation but not specifying whether it is a word or latent representation in Equation 1 - that only becomes obvious later. The exposition clearly needs more work, and more precise technical writing.
2. There is a lot of related work around word embeddings that is not mentioned, both on word2vec-style representation learning (e.g. it would be useful to relate this more to word2vec and what it learns, as in Omer Levy's work on "interpreting" word2vec, rather than reinterpreting) and word embeddings on hypernymy detection and lexical entailment (see e.g. Stephen Roller's thesis for references).
3. There has been a lot of work on the Weeds BLESS dataset that is not mentioned, or compared against, including unsupervised approaches (e.g. Levy's work, Santus's work, Kiela's work, Roller's work, etc.), that perform better than the numbers in Table 1. There are many other datasets that measure lexical entailment, none of which are evaluated on (apart from the original BLESS set, which is mentioned in passing). It would make sense to show that the method works on more than one dataset, and to do a thorough comparison against other work; especially given that:
4. The tone of the work appears to imply that word2vec was wrong and needs to be reinterpreted: the work leads to "unprecedented results" (not true), claims to be a completely novel method for inducing word representations (together with LSA, BOW and Word2Vec, third paragraph; not true), and suggests it has found "the best way to extract information about the semantics of a word from this model" (7th paragraph; not true). This, together with the "reinterpretation of word2vec" and the proposed "new distributional semantic models" almost makes it hard for me to take the work seriously.