DON'T SETTLE FOR AVERAGE, GO FOR THE MAX: FUZZY SETS AND MAX-POOLED WORD VECTORS
Recent literature suggests that averaged word vectors followed by simple postprocessing outperform many deep learning methods on semantic textual similarity tasks. Furthermore, when averaged word vectors are trained supervised on large corpora of paraphrases, they achieve state-of-the-art results on standard STS benchmarks. Inspired by these insights, we push the limits of word embeddings even further. We propose a novel fuzzy bag-of-words (FBoW) representation for text that contains all the words in the vocabulary simultaneously but with different degrees of membership, which are derived from similarities between word vectors. We show that max-pooled word vectors are only a special case of fuzzy BoW and should be compared via fuzzy Jaccard index rather than cosine similarity. Finally, we propose DynaMax, a completely unsupervised and non-parametric similarity measure that dynamically extracts and max-pools good features depending on the sentence pair. This method is both efficient and easy to implement, yet outperforms current baselines on STS tasks by a large margin and is even competitive with supervised word vectors trained to directly optimise cosine similarity.
This submission presents a simple model for sentence representation based on max-pooling of word vectors. The model is motivated by fuzzy-set theory, providing both a funded pooling scheme and a similarity score between documents. The proposed approach is evaluated on sentence similarity tasks (STS) and achieves very strong performance, comparable to state-of-the-art, computationally demanding methods.
Pros:
+ The problem tackled by this paper is interesting and well motivated. Fast, efficient and non-parametric sentence similarity has tons of important applications (search, indexing, corpus mining).
+ The proposed solution is elegant and very simple to implement. 
+ When compared to standard sentence representation models, the proposed approach has very good performance, while being very efficient. It only requires a matrix vector product and a dimension-wise max.
+ The paper is very well written and flows nicely.
+ Empirical results show significant differences between different word vectors. The simplicity of this approach makes it a good test bed for research on word vectors.
Cons:
- Nothing much, really. 
- Eq. (3) is awkward, as it is a sequence of equalities, which has to be avoided. Moreover, if U is the identity, I don't think that the reader really need this Eq...
I have several questions and remarks that, if answered would make the quality of the presentation better:
* In infersent, the authors reported the performance of a randomly-initialized and max-pooled bi-lstm with fasttext vectors as the input lookup. This can be seen as an extreme case of the presented formalism, where the linear operator U is replaced by a complicated non linear function that is implemented by the random LSTM. Drawing that link, and maybe including this baseline in the results would be good.
* Related to this previous question, several choices for U are discussed in the paper. However, only two are compared in the experiments. It would be interesting to have an experimental comparison of:
- taking U = W
- taking U = I
- taking U as the principal directions of W
- taking U as a random matrix, and comparing performance for different output dimensions. 
Overall, this paper is a very strong baseline paper. The presented model is elegant and efficient. I rate it as an 8 and await other reviews and the author's response.