Sentence encoders are typically trained on generative language modeling tasks with large unlabeled datasets. While these encoders achieve strong results on many sentence-level tasks, they are difficult to train with long training cycles. We introduce fake sentence detection as a new discriminative training task for learning sentence encoders. We automatically generate fake sentences by corrupting original sentences from a source collection and train the encoders to produce representations that are effective at detecting fake sentences. This binary classification task turns to be quite efficient for training sentence encoders. We compare a basic BiLSTM encoder trained on this task with strong sentence encoding models (Skip-thought and FastSent) trained on a language modeling task. We find that the BiLSTM trains much faster on fake sentence detection (20 hours instead of weeks) using smaller amounts of data (1M instead of 64M sentences). Further analysis shows the learned representations also capture many syntactic and semantic properties expected from good sentence representations.
Summary:
=======
The paper proposes a discriminative training formulation for learning sentence representations, where a classifier is required to distinguish between real and fake sentences. The sentences are encoded with a Bi-LSTM and the resulting sentence representations are then used in a number of sentence-level tasks (classification, entailment, and retrieval). The experiments show benefits on most tasks compared to Skip-Thought and FastSent baselines, and the information captured by the representations is analyzed with probing tasks, showing that they are better at capturing certain kinds of information like the presence or order of words. 
The paper proposes a simple and fairly effective approach for learning sentence encoders. The basic idea is appealing and the experimental results are fairly good. However, at present it seems like more work is required for delivering a comprehensive evaluation and analysis. My main concerns with the paper are the insufficient comparison with prior work, its lack of clarity and organization in certain places, and the limited amount of work. Please see below detailed comments on these and other points, as well as suggestions for how to improve some of these issues.  
Major comments:
==============
1. Better baselines and comparisons: 
- The results are compared only with SKip-Thought and (the weaker) FastSent. However, there are far better models by now. First, already in the Skip-Thought paper there is a version combining Naive Bayes bi-gram features which performs much better on some benchmarks, for example that version would be better than the paper's results on MR (80.4). 
- Moreover, there have been many newer papers with better results on many of the tasks [1, 2, 4, and references therein]. At the very least, mention should be made that there are better published results, and ideally there should be some comparison to the more relevant papers [1, and maybe others].  
2. Paper organization and clarity:
- I found Section 2 to be unnecessarily lengthy and disorganized. It mixes motivation with modeling, introduces excessive notation, sometimes without clearly defining it (what is L_{aux}? Why is U in eq. 2 not defined on first usage?), and digresses to weakly related discussions (the link to GANs seems vague and the relation to Introspective Neural Networks is not made clear). The last paragraph is largely redundant with the introduction. 
- There is also a statement that seems just wrong: "maximizing the data likelihood P(enc(x,\theta_1)|\theta_1,\thera_3)" -- the data likelihood is P(X | ...). Maximizing the encoding of x can be trivially achieved by simply having a constant encoding whose probability is 1. 
- The entire Section 2 can be condensed to one or two paragraph, essentially deriving the discriminative training task in equations (1) and (2). 
- On the paper organization level, this lengthy section is followed by the related work and then section 4 on "training tasks for encoders". There is again redundancy between section 4 and 2. Consider merging sections 2 and 4 into one Methodology section, where the general task is formulated, the sentence encoding (Bi-LSTM with max-pooling) and binary classifier (the MLP) are defined, and the fake sentence generation is described. This would make a better flow and remove excessive text. 
3. Motivation and advantages of the approach:
- The approach is motivated by shortcomings of sentence encodings based on language modeling, such as Skip-Thought, which are computationally intensive due to the large output space and the complicated decoding process. This is an appealing motivation, although there have also been simpler methods for sentence representations that work as well as or better than Skip-Thought [1, 2]. 
- The second motivation is not clear to me, and the claim that "the training text collection should include many instances of sentences that have only minor lexical differences but found in completely different contexts" needs more support, either theoretical or empirical. Why wouldn't a language model be able to distinguish such differences?
- The advantages of the binary classification task make sense. The point about forcing the encoder to track both syntax and semantics is interesting. Have you tried to analyze whether this indeed happens? The probing tasks are a good way to evaluate this, but most of them are syntactic, except SOMO and perhaps CoordInv and BShift. Still, more analysis of this point would be good. 
- One concern with generating fake sentences by swapping words is that it would not apply to languages with free word order. Have you considered how well your approach would work on other languages? 
4. Relevant related work: 
- The fake data generation resembles noise used in denoising auto-encoders. A recent application is in unsupervised neural machine translation [3], but there is relevant prior work (see references in [3]). 
- The binary classification task resembles that in [1], where they train a classifier to distinguish between the representation of a correct neighbor sentences and incorrect sentences. 
5. Ideas for more experiments and analysis:
- The results are fairly good by using only 1M sentences. How good would they be with the full corpus? What's the effect of training data size on the method? 
- Table 4 is providing nice examples showing how the fake sentence task generates better sentences representations. Can this be measured on a larger set of examples in aggregate? Why is t-SNE needed for calculating the neighbor rank? 
- Proving tasks are very interesting, but the discussion is limited. A more detailed discussion and analysis would be useful. 
- Consider other techniques for generating fake sentences. 
Minor comments:
==============
- Related work: the Skip-Thought decoder is a unidirectional LSTM and not a bidirectional one as mentioned, right? 
- Related work: more details on supervised approaches would be useful. 
- Section 4.1: how many fake exampels are generated from every real example? Have you experimented with this? 
- Section 4.2 mentions 2 hidden layers in the MLP but figure 3 indicates 3 layers. 
- Is there a reason to use multiple layers without a non-linearity in the MLP? This seems unusual. In terms of expressivity, this is equivalent to using one larger linear layer, although there might be some benefit in optimization. 
- Table 1 seems unnecessary as there is no discussion of how dataset statistics refer to the results. It's enough to refer to previous work. 
- What are some results missing in table 2, specifically SKipthought (1M) on COCO datasets? 
- The paragraph on sentence encoder implementation mentions a "validation set accuracy of 89 for word shuffle". Which validation set is that? How is convergence determined for word drop? 
- In analyzing sentence lengths, figure 2 shows the fake sentence to be similar to SKip-Thought on short sentences in SST. Do you have any idea why? Also, fake sentence is better than Skip-Thought on all lengths in MR, not just longer sentences, so I'm not sure there's any signal there. 
- Figure 3: what is the test set for WordShuffle? 
- The idea to create negative samples focused towards specific phenomena sounds like a good way to go
Writing, grammar, etc.:
======================
- Introduction, paragraph 3, last sentence: start with "The". 
- Introduction, paragraph 4, first sentences: discriminative training task fake sentence detection -> discriminative training task *of* fake sentence detection
- Motivation: an useful -> a useful; we assumes -> we assume; then number -> the number; this much -> this is much
- Motivation: do not differ -> do not differ much? 
- Related work: skip-gram -> skip-gram model; Training Skipthought model -> Training a Skipthought model
- Section 4: Prior work use -> Prior work uses/used 
- Section 4.2: space between "Multi-layer Perceptron" and "(MLP)". This also happens with other acronyms. 
- Page 6: Our models, however, train -> are trained 
- Table 3 caption: is bigram in -> is bigram; is co-ordination is -> is-coordination 
- Page 7: The analysis ... also indicate*s* ... but do*es* not ... 
- Figure 3 caption: classification/proving task -> tasks 
- References: fix capitalization in paper titles
References
==========
[1] Logeswaran and Lee, An efficient framework for learning sentence representations
[2] Khodak et al., A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors
[3] Artetxe et al., Unsupervised Neural Machine Translation
[4] Arora et al., A Compressed Sensing View of Unsupervised Text Embeddings, Bag-of-n-Grams, and LSTMs