We introduce a probabilistic framework for quantifying the semantic similarity between two groups of embeddings. We formulate the task of semantic similarity as a model comparison task in which we contrast a generative model which jointly models two sentences versus one that does not. We illustrate how this framework can be used for the Semantic Textual Similarity tasks using clear assumptions about how the embeddings of words are generated. We apply information criteria based model comparison to overcome the shortcomings of Bayesian model comparison, whilst still penalising model complexity. We achieve competitive results by applying the proposed framework with an appropriate choice of likelihood on the STS datasets.
The authors propose a probabilistic model for computing the sentence similarity between two sets of representations in an online fashion (that is, they do not need to see the entire dataset at once as SIF does when using PCA). They evaluate on the STS tasks and outperform competitive baselines like WMD, averaging embeddings, and SIF (without PCA), but they have worse performance that SIF + PCA.
The paper is clearly written and their model is carefully laid out along with their derivation. My concern with this paper however, is that I feel the paper lacks a motivation, was it derive an online similarity metric that outperforms SIF(without PCA)?
A few experimental questions/comments:
What happens to all methods when stop words are not removed? How far does performance fall? I think one reason it might fall (in addition to the reasons given in the paper) is that all vectors are set to have the same norm. For STS tasks, often the norms of these vectors are reduced during training which lessens their influence. What mechanism was used to identify the stop words and does removing these help the other methods (I know in the paper, stop words were removed in the baseline, did this unilaterally improve performance for these methods)?
Overall I do like the paper, however I do find the results to be lackluster. There are many papers on combining word embeddings trained in various ways that have much stronger numbers on STS, but these methods won't be effective with this type of similarity (namely because embeddings must have unit norm in their model). Therefore, I think the paper needs some more motivation and experimental evidence of its superiority over related methods like SIF+PCA in order for it to be accepted.
PROS
- Probabilistic model with clear design assumptions from which a similarity metric can be derived.
- Derived similarity metric doesn't require knowledge of the entire dataset (in comparison to SIF + PCA)
CONS
- Performance seems to be slightly better than SIF, WMD, and averaging word embeddings, but below that of SIF + PCA 
- Unclear motivation for the model, was it derive an online similarity metric that outperforms SIF(without PCA)?
- Requires the removal of stop words, but doesn't state how these were defined. Minor point, but tuning this could be enough to cause the improvement over related methods.