Context Mover*s Distance & Barycenters: Optimal transport of contexts for building representations
We propose a unified framework for building unsupervised representations of entities and their compositions, by viewing each entity as a histogram (or distribution) over its contexts. This enables us to take advantage of optimal transport and construct representations that effectively harness the geometry of the underlying space containing the contexts. Our method captures uncertainty via modelling the entities as distributions and simultaneously provides interpretability with the optimal transport map, hence giving a novel perspective for building rich and powerful feature representations. As a guiding example, we formulate unsupervised representations for text, and demonstrate it on tasks such as sentence similarity and word entailment detection. Empirical results show strong advantages gained through the proposed framework. This approach can potentially be used for any unsupervised or supervised problem (on text or other modalities) with a co-occurrence structure, such as any sequence data. The key tools at the core of this framework are Wasserstein distances and Wasserstein barycenters.
 The submission explores a new form of word representation based on a histogram over context word vectors, allowing them to measure distances between words in terms of optimal transport between these histograms. The authors speculate that this may allow better representations of polysemous words. The approach is mathematically elegant, and requires no additional training on top of existing approaches like Glove. To improve efficiency, they use clustering on context vectors. They present results on various semantic textual similarity and hypernym detection tasks, outperforming some baselines. The paper presents itself as an alternative to word embeddings as a way of representing words. As far as I can tell, their method only really allows a way of computing distances between pairs of word representations, which hasn*t been a useful concept for the vast majority of cases that word embeddings have been used for (translation, QA, etc.). Point estimates are at least very convenient to work with. That doesn*t mean the proposed approach is useless, but the paper needs to give a much stronger motivation for when and why measuring distances between words may be helpful. The experiments are a bit underwhelming. STS and hypernymy detection are somewhat unimpressive tasks to work - I*m not aware of any results on these tasks that have generalized to more realistic applications like translation or question answering. I think for publication with just these tasks, the method would need to show a dramatic breakthrough, which the submission definitely does not. The STS baselines are very simple bag-of-words approaches, and even then the results for the SIF baseline are much lower than those reported by Arora et al. (2017). At the least, there should be a comparison with the current state of the art. On the hypernymy task, what validation data was used? Unfortunately I*m not able to suggest better experiments, because I can*t think of cases where their method would be useful. The paper is significantly weakened by frequently making very strong claims based on rather limited experimental results (for one example, "we illustrate how our framework can be of significant benefit for a wide variety of important tasks" feels like quite a stretch). It would be much improved if some of the language was toned down. Overall, the paper introduces a mathematically elegant method for representing words as distributions over contexts, and for computing distances between these words. For acceptance, I think the paper needs to better motivate why the method could be useful, and back that up with more convincing experiments.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Context Mover*s Distance & Barycenters: Optimal transport of contexts for building representations
We propose a unified framework for building unsupervised representations of entities and their compositions, by viewing each entity as a histogram (or distribution) over its contexts. This enables us to take advantage of optimal transport and construct representations that effectively harness the geometry of the underlying space containing the contexts. Our method captures uncertainty via modelling the entities as distributions and simultaneously provides interpretability with the optimal transport map, hence giving a novel perspective for building rich and powerful feature representations. As a guiding example, we formulate unsupervised representations for text, and demonstrate it on tasks such as sentence similarity and word entailment detection. Empirical results show strong advantages gained through the proposed framework. This approach can potentially be used for any unsupervised or supervised problem (on text or other modalities) with a co-occurrence structure, such as any sequence data. The key tools at the core of this framework are Wasserstein distances and Wasserstein barycenters.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The submission explores a new form of word representation based on a histogram over context word vectors, allowing them to measure distances between words in terms of optimal transport between these histograms. The authors speculate that this may allow better representations of polysemous words. The approach is mathematically elegant, and requires no additional training on top of existing approaches like Glove. To improve efficiency, they use clustering on context vectors. They present results on various semantic textual similarity and hypernym detection tasks, outperforming some baselines. The paper presents itself as an alternative to word embeddings as a way of representing words. As far as I can tell, their method only really allows a way of computing distances between pairs of word representations, which hasn*t been a useful concept for the vast majority of cases that word embeddings have been used for (translation, QA, etc.). Point estimates are at least very convenient to work with. That doesn*t mean the proposed approach is useless, but the paper needs to give a much stronger motivation for when and why measuring distances between words may be helpful. The experiments are a bit underwhelming. STS and hypernymy detection are somewhat unimpressive tasks to work - I*m not aware of any results on these tasks that have generalized to more realistic applications like translation or question answering. I think for publication with just these tasks, the method would need to show a dramatic breakthrough, which the submission definitely does not. The STS baselines are very simple bag-of-words approaches, and even then the results for the SIF baseline are much lower than those reported by Arora et al. (2017). At the least, there should be a comparison with the current state of the art. On the hypernymy task, what validation data was used? Unfortunately I*m not able to suggest better experiments, because I can*t think of cases where their method would be useful. The paper is significantly weakened by frequently making very strong claims based on rather limited experimental results (for one example, "we illustrate how our framework can be of significant benefit for a wide variety of important tasks" feels like quite a stretch). It would be much improved if some of the language was toned down. Overall, the paper introduces a mathematically elegant method for representing words as distributions over contexts, and for computing distances between these words. For acceptance, I think the paper needs to better motivate why the method could be useful, and back that up with more convincing experiments.
assistant
 The submission explores a new form of word representation based on a histogram over context word vectors, allowing them to measure distances between words in terms of optimal transport between these histograms. The authors speculate that this may allow better representations of polysemous words. The approach is mathematically elegant, and requires no additional training on top of existing approaches like Glove. To improve efficiency, they use clustering on context vectors. They present results on various semantic textual similarity and hypernym detection tasks, outperforming some baselines. The paper presents itself as an alternative to word embeddings as a way of representing words. As far as I can tell, their method only really allows a way of computing distances between pairs of word representations, which hasn*t been a useful concept for the vast majority of cases that word embeddings have been used for (translation, QA, etc.). Point estimates are at least very convenient to work with. That doesn*t mean the proposed approach is useless, but the paper needs to give a much stronger motivation for when and why measuring distances between words may be helpful. The experiments are a bit underwhelming. STS and hypernymy detection are somewhat unimpressive tasks to work - I*m not aware of any results on these tasks that have generalized to more realistic applications like translation or question answering. I think for publication with just these tasks, the method would need to show a dramatic breakthrough, which the submission definitely does not. The STS baselines are very simple bag-of-words approaches, and even then the results for the SIF baseline are much lower than those reported by Arora et al. (2017). At the least, there should be a comparison with the current state of the art. On the hypernymy task, what validation data was used? Unfortunately I*m not able to suggest better experiments, because I can*t think of cases where their method would be useful. The paper is significantly weakened by frequently making very strong claims based on rather limited experimental results (for one example, "we illustrate how our framework can be of significant benefit for a wide variety of important tasks" feels like quite a stretch). It would be much improved if some of the language was toned down. Overall, the paper introduces a mathematically elegant method for representing words as distributions over contexts, and for computing distances between these words. For acceptance, I think the paper needs to better motivate why the method could be useful, and back that up with more convincing experiments.