While pre-training and fine-tuning, e.g., BERT (Devlin et al., 2018), GPT-2 (Radford et al., 2019), have achieved great success in language understanding and generation tasks, the pre-trained models are usually too big for online deployment in terms of both memory cost and inference speed, which hinders them from practical online usage. In this paper, we propose LightPAFF, a Lightweight Pre-training And Fine-tuning Framework that leverages two-stage knowledge distillation to transfer knowledge from a big teacher model to a lightweight student model in both pre-training and fine-tuning stages. In this way the lightweight model can achieve similar accuracy as the big teacher model, but with much fewer parameters and thus faster online inference speed. LightPAFF can support different pretraining methods (such as BERT, GPT-2 and MASS (Song et al., 2019)) and be applied to many downstream tasks. Experiments on three language understanding tasks, three language modeling tasks and three sequence to sequence generation tasks demonstrate that while achieving similar accuracy with the big BERT, GPT-2 and MASS models, LightPAFF reduces the model size by nearly 5x and improves online inference speed by 5x-7x.
Summary: This work leverages knowledge distillation both in pre-training and fine-tuning stages to learn a more compact student model that approximates the performance of a teacher model. Extensive experiments with different knowledge distillation loss functions are conducted on a number of representative language representation models including BERT, GPT-2 and MASS to show the benefit of the method. 
Strengths:
The numerical results show some promise of the proposed method to achieve competitive performance to the teacher network and outperform the considered baselines. The evaluated 9 downstream tasks cover a wide range of language understanding and generation problems and they do prove the effectiveness of knowledge distillation in boosting the accuracy.
Weaknesses:
My main concern is about the limited novelty of approach. The knowledge distillation method developed in the paper seems not particularly novel in principle. Although extensive numerical results are provided to show the effectiveness of knowledge distillation, the value-added beyond applying different distillation loss functions in different NLP tasks is still insufficient. Particularly, the loss function that combines maximal likelihood loss with KL divergence looks similar to those used in the paper [1]. The connection and difference to that related work need to be clarified. 
[1] Yu, Dong, et al. "KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition." 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2013.
=== update after author response ===
Thank you for the response. Although interesting and showing some promise in a bunch of applications, I am still not convinced that the proposed approach is novel enough in principle.  I thus stick to my assessment of this paper as borderline leaning to rejection.