Model Comparison for Semantic Grouping
We introduce a probabilistic framework for quantifying the semantic similarity between two groups of embeddings. We formulate the task of semantic similarity as a model comparison task in which we contrast a generative model which jointly models two sentences versus one that does not. We illustrate how this framework can be used for the Semantic Textual Similarity tasks using clear assumptions about how the embeddings of words are generated. We apply information criteria based model comparison to overcome the shortcomings of Bayesian model comparison, whilst still penalising model complexity. We achieve competitive results by applying the proposed framework with an appropriate choice of likelihood on the STS datasets.
 Main contribution: devising and evaluating a theoretically-sound algorithm for quantifying the semantic similarity between two pieces of text (e.g., two sentences), given pre-trained word embeddings (glove). Clarity: The paper is generally well-written, but I would have liked to see more details regarding the motivation for the work, description of the prior work and discussion of the results. As an example, I could not understand what were the differences between the online and offline settings, with only a reference to the (Arora et al. 2016) paper that does not contain neither "online" nor "offline". The mathematical derivations are detailed, which is nice. Originality: The work looks original. It proposes a method for quantifying semantic similarity that does not rely on cosine similarity. Significance: I should start by saying I am not a great reviewer for this paper. I am not familiar with the STS dataset and don*t have the mathematical background to fully understand the author*s algorithm. I like to see theoretical work in a field that desperately needs some, but overall I feel the paper could do a much better job at explaining the motivation behind the work, which is limited to "cosine similarity [...] is not backed by a solid theoretical foundation". I am not convinced of the practicality of the algorithm either: the algorithm seems to improve slightly over the compared approaches (and it is unclear if the differences are significant), and only in some settings. The approach needs to remove stop-words, which is reminiscent of good old feature engineering. Finally, the paper claims better average time complexity than some other methods, but discussing whether the algorithm is faster for common ranges of d (the word embedding dimension) would also have been interesting.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Model Comparison for Semantic Grouping
We introduce a probabilistic framework for quantifying the semantic similarity between two groups of embeddings. We formulate the task of semantic similarity as a model comparison task in which we contrast a generative model which jointly models two sentences versus one that does not. We illustrate how this framework can be used for the Semantic Textual Similarity tasks using clear assumptions about how the embeddings of words are generated. We apply information criteria based model comparison to overcome the shortcomings of Bayesian model comparison, whilst still penalising model complexity. We achieve competitive results by applying the proposed framework with an appropriate choice of likelihood on the STS datasets.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Main contribution: devising and evaluating a theoretically-sound algorithm for quantifying the semantic similarity between two pieces of text (e.g., two sentences), given pre-trained word embeddings (glove). Clarity: The paper is generally well-written, but I would have liked to see more details regarding the motivation for the work, description of the prior work and discussion of the results. As an example, I could not understand what were the differences between the online and offline settings, with only a reference to the (Arora et al. 2016) paper that does not contain neither "online" nor "offline". The mathematical derivations are detailed, which is nice. Originality: The work looks original. It proposes a method for quantifying semantic similarity that does not rely on cosine similarity. Significance: I should start by saying I am not a great reviewer for this paper. I am not familiar with the STS dataset and don*t have the mathematical background to fully understand the author*s algorithm. I like to see theoretical work in a field that desperately needs some, but overall I feel the paper could do a much better job at explaining the motivation behind the work, which is limited to "cosine similarity [...] is not backed by a solid theoretical foundation". I am not convinced of the practicality of the algorithm either: the algorithm seems to improve slightly over the compared approaches (and it is unclear if the differences are significant), and only in some settings. The approach needs to remove stop-words, which is reminiscent of good old feature engineering. Finally, the paper claims better average time complexity than some other methods, but discussing whether the algorithm is faster for common ranges of d (the word embedding dimension) would also have been interesting.
assistant
 Main contribution: devising and evaluating a theoretically-sound algorithm for quantifying the semantic similarity between two pieces of text (e.g., two sentences), given pre-trained word embeddings (glove). Clarity: The paper is generally well-written, but I would have liked to see more details regarding the motivation for the work, description of the prior work and discussion of the results. As an example, I could not understand what were the differences between the online and offline settings, with only a reference to the (Arora et al. 2016) paper that does not contain neither "online" nor "offline". The mathematical derivations are detailed, which is nice. Originality: The work looks original. It proposes a method for quantifying semantic similarity that does not rely on cosine similarity. Significance: I should start by saying I am not a great reviewer for this paper. I am not familiar with the STS dataset and don*t have the mathematical background to fully understand the author*s algorithm. I like to see theoretical work in a field that desperately needs some, but overall I feel the paper could do a much better job at explaining the motivation behind the work, which is limited to "cosine similarity [...] is not backed by a solid theoretical foundation". I am not convinced of the practicality of the algorithm either: the algorithm seems to improve slightly over the compared approaches (and it is unclear if the differences are significant), and only in some settings. The approach needs to remove stop-words, which is reminiscent of good old feature engineering. Finally, the paper claims better average time complexity than some other methods, but discussing whether the algorithm is faster for common ranges of d (the word embedding dimension) would also have been interesting.