Looking for ELMo*s friends: Sentence-Level Pretraining Beyond Language Modeling
Work on the problem of contextualized word representation—the development of reusable neural network components for sentence understanding—has recently seen a surge of progress centered on the unsupervised pretraining task of language modeling with methods like ELMo (Peters et al., 2018). This paper contributes the first large-scale systematic study comparing different pretraining tasks in this context, both as complements to language modeling and as potential alternatives. The primary results of the study support the use of language modeling as a pretraining task and set a new state of the art among comparable models using multitask learning with language models. However, a closer look at these results reveals worryingly strong baselines and strikingly varied results across target tasks, suggesting that the widely-used paradigm of pretraining and freezing sentence encoders may not be an ideal platform for further work.
 Only a handful of NLP tasks have an ample amount of labeled data to get state-of-the-art results without using any form of transfer learning. Training sentence representation in an unsupervised manner is hence crucial for real-world NLP applications. Contextualized word representations have gained a lot of interest in recent years and the NLP and ML community could benefit from such detailed comparison of such methods. This paper*s biggest strength is the experimental setting. The authors cover a lot of ground in comparing a lot of the recent work, both qualitatively and quantitatively -- there are a lot of experiments. I do understand the computational limitations of the authors (as they mention on HYPERPARAMETER TUINING) and I do agree with their statement “ The choice not to tune limits our ability to diagnose the causes of poor performance when it occurs”. Extensive hyper-parameter tuning can make a substantial different when dealing with NN models, maybe the authors should have considered dropping some of the tasks (the article has more than enough IMHO) and focus on a smaller sub set of tasks with proper hyper-parameter tuning. Table 2 is very interesting, the results suggesting that we are indeed very far from fully robust sentence representation method.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Looking for ELMo*s friends: Sentence-Level Pretraining Beyond Language Modeling
Work on the problem of contextualized word representation—the development of reusable neural network components for sentence understanding—has recently seen a surge of progress centered on the unsupervised pretraining task of language modeling with methods like ELMo (Peters et al., 2018). This paper contributes the first large-scale systematic study comparing different pretraining tasks in this context, both as complements to language modeling and as potential alternatives. The primary results of the study support the use of language modeling as a pretraining task and set a new state of the art among comparable models using multitask learning with language models. However, a closer look at these results reveals worryingly strong baselines and strikingly varied results across target tasks, suggesting that the widely-used paradigm of pretraining and freezing sentence encoders may not be an ideal platform for further work.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Only a handful of NLP tasks have an ample amount of labeled data to get state-of-the-art results without using any form of transfer learning. Training sentence representation in an unsupervised manner is hence crucial for real-world NLP applications. Contextualized word representations have gained a lot of interest in recent years and the NLP and ML community could benefit from such detailed comparison of such methods. This paper*s biggest strength is the experimental setting. The authors cover a lot of ground in comparing a lot of the recent work, both qualitatively and quantitatively -- there are a lot of experiments. I do understand the computational limitations of the authors (as they mention on HYPERPARAMETER TUINING) and I do agree with their statement “ The choice not to tune limits our ability to diagnose the causes of poor performance when it occurs”. Extensive hyper-parameter tuning can make a substantial different when dealing with NN models, maybe the authors should have considered dropping some of the tasks (the article has more than enough IMHO) and focus on a smaller sub set of tasks with proper hyper-parameter tuning. Table 2 is very interesting, the results suggesting that we are indeed very far from fully robust sentence representation method.
assistant
 Only a handful of NLP tasks have an ample amount of labeled data to get state-of-the-art results without using any form of transfer learning. Training sentence representation in an unsupervised manner is hence crucial for real-world NLP applications. Contextualized word representations have gained a lot of interest in recent years and the NLP and ML community could benefit from such detailed comparison of such methods. This paper*s biggest strength is the experimental setting. The authors cover a lot of ground in comparing a lot of the recent work, both qualitatively and quantitatively -- there are a lot of experiments. I do understand the computational limitations of the authors (as they mention on HYPERPARAMETER TUINING) and I do agree with their statement “ The choice not to tune limits our ability to diagnose the causes of poor performance when it occurs”. Extensive hyper-parameter tuning can make a substantial different when dealing with NN models, maybe the authors should have considered dropping some of the tasks (the article has more than enough IMHO) and focus on a smaller sub set of tasks with proper hyper-parameter tuning. Table 2 is very interesting, the results suggesting that we are indeed very far from fully robust sentence representation method.