Word embedding plays a key role in various tasks of natural language processing. is carried with the resulting embeddings. To generate interpretable word embed-3 dings we intend to replace the word vector with a probability density distribution.
The paper addresses the problem the problem in word embeddings where "word-word" or "context-context" inner products are relied upon in practice when the embeddings are usually only optimized for good properties of "word-context" inner products. The new approach to training word embeddings addresses interpretability in the sense of having a theoretically grounded interpretation for the good properties that the inner products should possess -- namely that they capture the PMI between words. By changing the embedding training procedure (to include a step that samples vectors from each word's distribution representation) this interpretation is made possible without problematic theoretical assumptions. Now trained for the purpose they are intended to be used for, the approach yields strong SOTA results on the non-small challenge datasets.
The word is well exceptionally well motivated (we should directly optimize for the properties we want!) and situated in the literature. The opposition of Arora 2016 (getting at "interpretable" word embeddings with a latent variable model) and Mimno & Thompson 2017 (problematic assumptions in skip-gram embeddings) is particularly convincing. The vMF distribution is underexamined as tool for analyzing high-dimensional semantic vector distributions, and it is an excellent fit for the purposes of this project.
To back up claims of theoretical interpretability, a derivation (building on Ma 2017) proceeds without problematic leap thanks to a property introduced by a carefully selected (if straightforward) regularization term. This reviewer did not read the appendix (not sure where this would be located -- first time reviewing with OpenReview here), but the intuition doesn't seem to be much different than Ma.
To back up claims of applied performance, appropriate experiments are conducted on multiple evaluation datasets. The results seem to be fairly interpreted.
This reviewer decides to accept this paper for its balance of theoretical and empirical contributions and for the role it might play in reducing dependence on mysticisms in word embeddings (relying on accidental / uncharacterized properties of previous embedding strategies).
Suggestions for improvement:
- Run a spell checker. It will catch a large number of problems that weren't ever big enough to hurt my appreciation of the paper.
- Consider the creation of an adversarial dataset (of ~3 words in ~3 contexts) where techniques that optimize for the wrong thing will catastrophically fail but the proposed approach succeeds.
- Write one or two more sentences about the fate of \kappa_c -- is it ever updated or am I just missing something? Making a bigger point about leaving \kappa un-optimized shows there is room for additional depth in this line of thinking. (If you start learning concentration parameters, check out the Kent distribution for an even more expressive distribution on the unit hypersphere: https://en.wikipedia.org/wiki/Kent_distribution)
- Figure out what happened to your appendix.
- Watch out that readers/reviewers of similar work may try to read the word "interpretable" in the title as a reference to the much broader topic of interpretability in ML related to explaining a model's behavior. Is there a synonym for "interpretable" that won't raise this false link?