Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training. Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019). However, surprisingly, the simple baseline of just pre-training and fine-tuning compact models has been overlooked. In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work. Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation. The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements. Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data. One surprising observation is that they have a compound effect even when sequentially applied on the same data. To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.
The authors investigate the problem of training compact pre-trained language model via distillation. Their method consists of three steps: 1. pre-train the compact model LM 2. distill the compact model LM with a larger model (teacher) 3. fine-tune the compact model on target task This idea is not significantly new since it is quite common to apply distillation to compress models, and the results are largely empirical. From Table 3 the results on test sets are better than previous works, but not by much. The authors spend quite a of space on ablation studies to investigate the contribution of different factors, and on cross-domain transfers. They do manage to show that using a teacher for distilling a compact student model does better than directly pre-training a compact model on the NLI* task in section 6.3. It would be better if they could show it for other tasks on the benchmark as well. Overall I think this work is somewhat incremental, and falls below the acceptance threshold.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training. Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019). However, surprisingly, the simple baseline of just pre-training and fine-tuning compact models has been overlooked. In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work. Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation. The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements. Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data. One surprising observation is that they have a compound effect even when sequentially applied on the same data. To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors investigate the problem of training compact pre-trained language model via distillation. Their method consists of three steps: 1. pre-train the compact model LM 2. distill the compact model LM with a larger model (teacher) 3. fine-tune the compact model on target task This idea is not significantly new since it is quite common to apply distillation to compress models, and the results are largely empirical. From Table 3 the results on test sets are better than previous works, but not by much. The authors spend quite a of space on ablation studies to investigate the contribution of different factors, and on cross-domain transfers. They do manage to show that using a teacher for distilling a compact student model does better than directly pre-training a compact model on the NLI* task in section 6.3. It would be better if they could show it for other tasks on the benchmark as well. Overall I think this work is somewhat incremental, and falls below the acceptance threshold.
assistant
The authors investigate the problem of training compact pre-trained language model via distillation. Their method consists of three steps: 1. pre-train the compact model LM 2. distill the compact model LM with a larger model (teacher) 3. fine-tune the compact model on target task This idea is not significantly new since it is quite common to apply distillation to compress models, and the results are largely empirical. From Table 3 the results on test sets are better than previous works, but not by much. The authors spend quite a of space on ablation studies to investigate the contribution of different factors, and on cross-domain transfers. They do manage to show that using a teacher for distilling a compact student model does better than directly pre-training a compact model on the NLI* task in section 6.3. It would be better if they could show it for other tasks on the benchmark as well. Overall I think this work is somewhat incremental, and falls below the acceptance threshold.