Computing universal distributed representations of sentences is a fundamental task in natural language processing. We propose ConsSent, a simple yet surprisingly powerful unsupervised method to learn such representations by enforcing consistency constraints on sequences of tokens. We consider two classes of such constraints -sequences that form a sentence and between two sequences that form a sentence when merged. We learn sentence encoders by training them to distinguish between consistent and inconsistent examples, the latter being generated by randomly perturbing consistent examples in six different ways. Extensive evaluation on several transfer learning and linguistic probing tasks shows improved performance over strong unsupervised and supervised baselines, substantially surpassing them in several cases. Our best results are achieved by training sentence encoders in a multitask setting and by an ensemble of encoders trained on the individual tasks.
== Clarity == 
The primary strength of this paper is the simplicity of the approach.
Main idea #1: corrupt sentences (via random insertions/deletions/permutations), and train a sentence encoder to determine whether a sentence has been corrupted or not.
Main idea #2: split a sentence into two parts (two different ways to do this were proposed). Train a sequence encoder to encode each part such that we can tell whether the two parts came from the same sentence or not.
I can see that this would be very easy for others to implement, perhaps encouraging its adoption.
== Quality of results ==
The proposed approach is evaluated on the well-known SentEval benchmark.
It generally does not outperform supervised approaches such as InferSent and MultiTask. However, this is fine because the proposed approach uses no supervised data, and can be applied in domains/languages where supervised data is not available.
The approach is competitive with existing state-of-the-art sentence representations such as QuickThoughts. However, it is not definitively better:
Out of the 9 tasks with results for QuickThoughts, this approach (ConsSent) performs better on 3 (MPQA +0.1%, TREC +0.4%, MRPC +0.4%). For the other 6 tasks, ConsSent performs worse (MR -1.8%, CR -1.7%, SUBJ -1%, SST -3.8%, SK-R, -2.4%). Taken together, the losses seem to be larger than the gains.
Furthermore, the QuickThoughts results were obtained with a single model across all SentEval tasks. In contrast, the ConsSent approach requires a different hyperparameter setting for each task in order to achieve comparable results -- there is no single hyperparameter setting that would give state-of-the-art results across all tasks.
The authors also evaluate on the newly-released linguistic probing tasks in SentEval. They strongly outperform several existing methods on this benchmark. However, it is unclear why they did not compare against QuickThoughts, which was the strongest baseline on the original SentEval tasks.
== Originality ==
The proposed approach is simple and straightforward. This is on the whole a great thing, but perhaps not especially surprising from an originality/novelty perspective.
Therefore, the significance and impact of this approach really needs to be carried by the quality of the empirical results.
The sentence pair based approaches (ConsSent-N and C) are conceptually interesting, but don't seem to be responsible for the best results on the linguistic probing tasks.
== Conclusion ==
Pros:
- conceptual simplicity
- competitive results (better than many previous unsup. sentence representation methods, excluding QuickThoughts)
- strong results on SentEval's linguistic probing task
Cons:
- no single hyperparameter value (perturbation method and value for k) gets great results across all tasks
- some important baselines possibly missing for linguistic probing tasks