Unsupervised Learning of Sentence Representations Using Sequence Consistency
Computing universal distributed representations of sentences is a fundamental task in natural language processing. We propose ConsSent, a simple yet surprisingly powerful unsupervised method to learn such representations by enforcing consistency constraints on sequences of tokens. We consider two classes of such constraints – sequences that form a sentence and between two sequences that form a sentence when merged. We learn sentence encoders by training them to distinguish between consistent and inconsistent examples, the latter being generated by randomly perturbing consistent examples in six different ways. Extensive evaluation on several transfer learning and linguistic probing tasks shows improved performance over strong unsupervised and supervised baselines, substantially surpassing them in several cases. Our best results are achieved by training sentence encoders in a multitask setting and by an ensemble of encoders trained on the individual tasks.
 The paper presents an unsupervised sentence encoding method based on automatically generating inconsistent sentences by applying various transformations either to a single sentence or a pair and then training a model to classify the original sentences from the transformed ones. Overall, I like the paper as it presents a simple method for training unsupervised sentence models which then can be used as part of further NLP tasks. A few comments on the method and results: - The results on Table 2 shows that supervised methods outperform unsupervised methods as well as the consistency based models with MultiTask having the largest margin. It would*ve been interesting to experiment with training multi-task layers on top of the sentence encoder and see how it would*ve performed. - The detail of the architecture is slightly missing in a sense that it*s not directly clear from the text if the output of the BiLSTMs is the final sentence encoding or the final layer before softmax? - Also I would*ve thought that the output of LSTMs passed through nonlinear dense layers but the text refers to two linear layers. - When I first read the paper, my eyes were looking for the result when you combine all of the transformations and train a single model :) - any reason why you didn*t try this experiment? - The paper is missing comparison and reference to recent works on universal language models (e.g. Radford et al 2018, Peters et al 2018, Howard et al 2018) as they rely on more elaborate model architectures and training compared to this paper but ultimately you can use them as sentence encoders. - One final note, which could be a subsequent paper is to treat these transformations as part of an adversarial setup to further increase the robustness of a language model such as those mentioned previously.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Unsupervised Learning of Sentence Representations Using Sequence Consistency
Computing universal distributed representations of sentences is a fundamental task in natural language processing. We propose ConsSent, a simple yet surprisingly powerful unsupervised method to learn such representations by enforcing consistency constraints on sequences of tokens. We consider two classes of such constraints – sequences that form a sentence and between two sequences that form a sentence when merged. We learn sentence encoders by training them to distinguish between consistent and inconsistent examples, the latter being generated by randomly perturbing consistent examples in six different ways. Extensive evaluation on several transfer learning and linguistic probing tasks shows improved performance over strong unsupervised and supervised baselines, substantially surpassing them in several cases. Our best results are achieved by training sentence encoders in a multitask setting and by an ensemble of encoders trained on the individual tasks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper presents an unsupervised sentence encoding method based on automatically generating inconsistent sentences by applying various transformations either to a single sentence or a pair and then training a model to classify the original sentences from the transformed ones. Overall, I like the paper as it presents a simple method for training unsupervised sentence models which then can be used as part of further NLP tasks. A few comments on the method and results: - The results on Table 2 shows that supervised methods outperform unsupervised methods as well as the consistency based models with MultiTask having the largest margin. It would*ve been interesting to experiment with training multi-task layers on top of the sentence encoder and see how it would*ve performed. - The detail of the architecture is slightly missing in a sense that it*s not directly clear from the text if the output of the BiLSTMs is the final sentence encoding or the final layer before softmax? - Also I would*ve thought that the output of LSTMs passed through nonlinear dense layers but the text refers to two linear layers. - When I first read the paper, my eyes were looking for the result when you combine all of the transformations and train a single model :) - any reason why you didn*t try this experiment? - The paper is missing comparison and reference to recent works on universal language models (e.g. Radford et al 2018, Peters et al 2018, Howard et al 2018) as they rely on more elaborate model architectures and training compared to this paper but ultimately you can use them as sentence encoders. - One final note, which could be a subsequent paper is to treat these transformations as part of an adversarial setup to further increase the robustness of a language model such as those mentioned previously.
assistant
 The paper presents an unsupervised sentence encoding method based on automatically generating inconsistent sentences by applying various transformations either to a single sentence or a pair and then training a model to classify the original sentences from the transformed ones. Overall, I like the paper as it presents a simple method for training unsupervised sentence models which then can be used as part of further NLP tasks. A few comments on the method and results: - The results on Table 2 shows that supervised methods outperform unsupervised methods as well as the consistency based models with MultiTask having the largest margin. It would*ve been interesting to experiment with training multi-task layers on top of the sentence encoder and see how it would*ve performed. - The detail of the architecture is slightly missing in a sense that it*s not directly clear from the text if the output of the BiLSTMs is the final sentence encoding or the final layer before softmax? - Also I would*ve thought that the output of LSTMs passed through nonlinear dense layers but the text refers to two linear layers. - When I first read the paper, my eyes were looking for the result when you combine all of the transformations and train a single model :) - any reason why you didn*t try this experiment? - The paper is missing comparison and reference to recent works on universal language models (e.g. Radford et al 2018, Peters et al 2018, Howard et al 2018) as they rely on more elaborate model architectures and training compared to this paper but ultimately you can use them as sentence encoders. - One final note, which could be a subsequent paper is to treat these transformations as part of an adversarial setup to further increase the robustness of a language model such as those mentioned previously.