NO TRAINING REQUIRED: EXPLORING RANDOM EN- CODERS FOR SENTENCE CLASSIFICATION
We explore various methods for computing sentence representations from pretrained word embeddings without any training, i.e., using nothing but random parameterizations. Our aim is to put sentence embeddings on more solid footing by 1) looking at how much modern sentence embeddings gain over random methods-as it turns out, surprisingly little; and by 2) providing the field with more appropriate baselines going forward-which are, as it turns out, quite strong. We also make important observations about proper experimental protocol for sentence classification evaluation, together with recommendations for future research.
This paper is about exploring better baselines for sentence-vector representations through randomly initialized/untrained networks. I applaud the overall message of this paper that we need to evaluate our models more thoroughly and have better baselines. The experimentation is quite thorough and I like that you
1) explored several different architectures
2) varied the dimensionality of representations
3) examine representations with probing tasks in the Analysis section. 
Main Critique
- In your takeaways you say that, “For some of the benchmark datasets, differences between random and trained encoders are so small that it would probably be best not to use those tasks anymore.” I don’t think this follows from your results. Just because current trained encoders do not perform better than random encoders on these tasks doesn’t in itself mean these tasks aren’t good evaluation tasks. These tasks could be faulty for other reasons, but just because we have no better technique than random encoders currently, doesn’t make these evaluation tasks not worthwhile. Perhaps you could further examine what features (n-gram, etc.) it takes to do well on these tasks in order to argue that they shouldn’t be used.
- In your related work section you say that “We show that a lot of information may be crammed into vectors using randomly parameterized combinations of pre-trained word embeddings: that is, most of the power in modern NLP systems is derived from having high-quality word embeddings, rather than from having better encoders.” Did you run experiments with randomly initialized embeddings? This paper (https://openreview.net/forum?id=ryeNPi0qKX) finds that representations from LSTMs with randomly initialized embeddings can perform quite well on some transfer tasks. I think in order to make such a claim about the power of high-quality word embeddings you should include numbers comparing them to randomly initialized embeddings.
Questions
- Did you find that your results were sensitive to the initialization technique used for your random LSTMs / projections?
- Do you have a sense of why random non-linear features are able to perform well on these tasks? What kind of features are the skip-thought and InferSent representations learning if they do not perform much better? It’s interesting that many of the random encoder methods outperform the trained models on word content. I think you could discuss these Analysis section findings more.
Other Critiques
- In the introduction, instead of simply describing what is commonly done to obtain and evaluate sentence embeddings, it would be better to include a sentence or two about the motivation for sentence embeddings at all.
- The first sentence, “Sentence embeddings are learned non-linear recurrent combinations of pre-trained word embeddings”, doesn’t seem to be true as BOE representations are also sentence embeddings and CNNs/transformers could also work. “Non-linear” and “recurrent” are not inherent requirements for sentence embeddings, but just techniques that researchers commonly use.
- In the second paragraph of introduction instead saying “Natural language processing does not yet have a clear grasp on the relationship between word and sentence embeddings…” it might be better to say “NLP researchers” or the “NLP community” instead of “NLP” as a field doesn’t have a clear grasp.
- In the introduction: “It is unclear how much sentence-encoding architectures improve over the raw word embeddings, and what aspect of such architectures is responsible for any improvement.” It would be also good to mention that it’s unclear how much the training task / procedure also is affects improvements.
- You could describe more about applications of reservoir computing in your related work section as it’s been used in NLP before.
- I don’t think you actually ever describe the type of data that InferSent is trained on, only that it is “expensive” annotated data. It might be useful to add a sentence about natural language inference for clarity.
- In the conclusion, change “performance improvements are less than 1 and less than 2 points on average over the 10 SentEval tasks, respectively” to  “performance improvements are less than 2 percentage points on average over the 10 SentEval tasks, respectively”
- It would be nice if you bolded/underlined the best performing numbers in your results tables.