Deep, heavily overparameterized language models such as BERT, XLNet and T5 have achieved impressive success in many NLP tasks. However, their high model complexity requires enormous computation resources and extremely long training time for both pre-training and fine-tuning. Many works have studied model compression on large NLP models, but only focus on reducing inference cost/time , while still requiring  expensive training process. Other works use extremely large batch sizes to shorten the pre-training time  at the expense of high demand for computation resources . In this paper, inspired by the Early-Bird Lottery Tickets  studied for computer vision tasks, we propose EarlyBERT, a general computationally-efficient training algorithm applicable to both pre-training and fine-tuning of large-scale language models. We are the first to identify structured winning tickets in the early stage of BERT training , and use them for efficient training. Comprehensive pre-training and fine-tuning experiments on GLUE and SQuAD downstream tasks  show that EarlyBERT easily achieves comparable performance to standard BERT  with 35~45\% less training time. 
Heavily overparameterized language models such as BERT, XLNet and T5 have achieved impressive success in many NLP tasks. However, their high model complexity requires enormous computation resources and extremely long training time for both pre-training and fine-tuning. Many works have studied model compression on large NLP models, but only focusing on reducing inference time while still requiring an expensive training process. Other works use extremely large batch sizes to shorten the pre-training time , at the expense of higher computational resource demands . In this paper, inspired by the Early-Bird Lottery Tickets recently studied for computer vision tasks, we propose EarlyBERT, a general computationally-efficient training algorithm applicable to both pre-training and fine-tuning of large-scale language models. By slimming the self-attention and fully-connected sub-layers inside a transformer, we are the first to identify structured winning tickets in the early stage of BERT training . We apply those tickets towards efficient BERT training, and conduct comprehensive pre-training and fine-tuning experiments on GLUE and SQuAD downstream tasks . Our results show that EarlyBERT achieves comparable performance to standard BERT , with 35~45\% less training time. Code is available at URL
R
Deep, heavily
Heavily
R
focus
focusing
R
cost/time ,
time
A
an
A
,
R
high demand for computation resources
higher computational resource demands
A
recently
R
We
By slimming the self-attention and fully-connected sub-layers inside a transformer, we
R
, and use them for efficient training. Comprehensive
. We apply those tickets towards efficient BERT training, and conduct comprehensive
A
. Our results
D
easily
A
,
A
Code is available at URL