bert2BERT: Towards Reusable Pretrained Language Models Abstract In recent years, researchers tend to pre-train ever-larger language models to explore the upper limit of deep models. However, large language model pre-training costs intensive computational resources, and most of the models are trained from scratch without reusing the existing pre-trained models, which is wasteful. In this paper, we propose bert2BERT 1 , which can effectively transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and significantly improve the pre-training efficiency of the large model. Specifically, we extend the previous function-preserving (Chen et al., 2016) method proposed in computer vision on the Transformer-based language model, and further improve it by proposing a novel method, advanced knowledge for the large model's initialization. In addition, a two-stage learning method is proposed to further accelerate the pre-training. We conduct extensive experiments on representative PLMs (e.g., BERT and GPT) and demonstrate that (1) our method can save a significant amount of training cost compared with baselines including learning from scratch, StackBERT (Gong et al., 2019) and MSLT (Yang et al., 2020); (2) our method is generic and applicable to different types of pretrained models. In particular, bert2BERT saves about 45% and 47% computational cost of pretraining BERTBASE and GPTBASE by reusing the models of almost their half sizes. Pre-trained language models (PLMs), such as BERT (Devlin et al., 2019), GPT (Radford et al., 2018, 2019; Brown et al., 2020), ELECTRA (Clark et al., 2020), XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019), have achieved great success in natural language processing (NLP). However, the pre-training process of large PLMs can be extremely computationally expensive and produces huge carbon footprints. For example, GPT-3 uses 3.1E+6 GPU hours for training, at an estimated cost of $4.6 million 2 , consuming a lot of computing resources. Therefore, how to reduce the training cost of PLM is of great importance to Green AI (Schwartz et al., 2020). Recently, there is a trend of training extremely large models to explore the upper limits of PLMs. For example, large pre-trained models, including GPT-3 (Brown et al., 2020) (175B), PanGu(Zeng et al., 2021) (200B) and Switch Transformers (Fedus et al., 2021) (1571B), have been proved promising in language understanding and generation. However, these models are all pre-trained from scratch independently without utilizing the knowledge of smaller ones that have already been trained. On the other hand, our empirical studies show that the pre-trained models of different scales could share similar knowledge, for example in Figure 2, the attention patterns of the two PLMs with different sizes are similar. 2 https://lambdalabs.com/blog/ demystifying-gpt-3/ propose the bert2BERT method, which can efficiently transfer the learned knowledge of the smaller model to the large model. bert2BERT consists of two components: (1) For parameter initialization, we first extend the function preserving training (Chen et al., 2016) to PLMs by duplicating and stacking the parameters of the existing smaller PLM, which we call function-preserving initialization (FPI). FPI ensures that the initialized large model has almost the same behavior as the small model, so that the large model has a good starting point for later optimization. We also find that duplicating the weights of the upper layer to the current layer can further accelerate the convergence of the large model, which we call advanced knowledge initialization (AKI). Although the AKI somewhat violates the principle of function preserving, we find that empirically it also has a good starting point as shown in Table 1, which leads to a faster convergence rate and achieves higher training efficiency. (2) Secondly, a two-stage training strategy is further applied to the large model to accelerate the training process. To demonstrate the superiority of our method, we conduct extensive experiments on two representative PLMs: BERT and GPT, with different source model sizes. The results show that: (1) our method can save a significant amount of computation in pre-training compared to the traditional way of learning from scratch and progressive stacking methods such as StackBERT (Gong et al., 2019) and MSLT (Yang et al., 2020); (2) our method is model-agnostic, which can be applied on a wide range of Transformer-based PLMs. One typical example is that, when using a small pre-trained model with half the size of BERTBASE for initialization, bert2BERT saves 45% computation cost of the original BERTBASE pre-training. In general, our contributions are summarized as follows: (1) We explore a new direction for the efficient pre-training by reusing the trained parameters of small models to initialize the large model; (2) We successfully extend function preserving method (Chen et al., 2016) on BERT and further propose advanced knowledge initialization, which can effectively transfer the knowledge of the trained small model to the big model and improve the pre-training efficiency; (3) The proposed method outperforms other training methods and achieves 45% computation reduction on BERTBASE ; (4) Our method is generic, effective for both the BERT and GPT models, and have great potential to become an energy-efficient solution for pre-training super large-scale language models. Efficient Pre-training in NLP. The efficiency of pre-training has been explored by previous work. Some works (Gong et al., 2019; Yang et al., 2020; Gu et al., 2021) propose progressive learning to accelerate the pre-training, which are motivated by the fact that different layers have some similar knowledge (e.g., attention patterns). They start pre-training a small model with fewer Transformer layers, and then iteratively expand the model by stacking the already trained layers on the top. Another line of work proposes to back distill the knowledge of the small models into large models, which is termed as knowledge inheritance (Qin et al., 2021). Some works focus on the data effi-2135 ciency (Wu et al., 2021) and take notes for rare words during the pre-training process to help the model understand them when they occur next. ELECTRA (Clark et al., 2020) proposes a task of replaced token detection to predict whether each token in the input was replaced or not, which improves the pre-training efficiency. Our method is orthogonal to this kind of work and the combination of ELECTRA and bert2BERT could achieve better efficiency. In addition, there are several other orthogonal techniques for efficient pre-training: mixed-precision training (Shoeybi et al., 2019), large batch optimization (You et al., 2020), model architecture innovation (Lan et al., 2020), layer dropping technique (Zhang and He, 2020), etc. Reusable Neural Network. Reusable neural network, a topic related to transfer learning (Pan and Yang, 2010), is introduced to accelerate the model training in computer vision. One classical work is Net2Net (Chen et al., 2016), which first proposes the concept of the function-preserving transformation to make neural networks reusable. However, Net2Net randomly selects the neurons to be split. To handle this problem, some works (Wu et al., 2019, 2020b; Wang et al., 2019b; Wu et al., 2020a) leverage a functional steepest descent idea to decide the optimal subset of neurons to be split. The pruning technique (Han et al., 2015) is also introduced for reusable neural networks (Feng and Panda, 2020). In this paper, we study the reusable pre-trained language model and propose a new method, bert2BERT, to accelerate the pre-training of BERT and GPT. BERT consists of one embedding layer and multiple Transformer (Vaswani et al., 2017) layers. The embedding layer first maps the tokens in a sentence into vectors with an embedding matrix WE . Then one normalization layer is employed to produce the initial hidden states H 0 . The hidden states are iteratively processed by multiple Transformer layers as follows: H l = Transformer l ( H l  1 ) , l  [1 , L ] (1) where L denotes the number of Transformer layers, each including a multi-head attention (MHA) and a feed-forward network (FFN). MHA. It is composed of multiple parallel self-attention heads. The hidden states of the previous layer are fed into each head and then the outputs of all heads are summed to obtain the final output as follows: Q i , K i , V i = H l  1 W Ql,i , H l  1 W Kl,i , H l  1 W Vl,i , HHEAD l,i = softmax( Q i K iT  d k ) V i W Ol,i , MHA( H l  1 ) = a (cid:88) i =1 HHEAD l,i , HMHA l = LayerNorm( H l  1 + MHA( H l  1 )) . H l  1 is linearly projected to queries ( Q i ), keys ( K i ) and values ( V i ) using W Ql,i , W Kl,i , W Vl,i respectively. HHEAD l,i indicates the context-aware vector which is obtained by the scaled dot-product of queries and keys in the i -th attention head. a represents the number of self-attention heads. d k is the head dimension acting as the scaling factor. FFN. It consists of two linear layers and one GeLU activation function (Hendrycks and Gimpel, 2016), that is: HFFN l = GeLU( HMHA l W 1 l + b 1 l ) W 2 l + b 2 l , H l = LayerNorm( HMHA l + HFFN l ) . Layer Normalization. Both the modules of MHA and FFN have one layer normalization (Ba et al., 2016) that stabilizes the dynamics of the hidden state in the Transformer. Formally, it is written as: LayerNorm( H ) = ( H   H  H )  WLN + b LN , (4) where  means the element-wise multiplication. We aim to accelerate the pre-training of target model T ( L t , D t ) by transferring the knowledge of an existing pre-trained source model S ( L s , D s ) , where L s | t means the numbers of Transformer layer and D s | t means the model width (i.e., hidden size), satisfying L s  L t and D s  D t . Formally, our problem is two-fold: (1) how to perform an effective parameter initialization for T by reusing the trained parameters of S , and (2) how to efficiently 2136 train the initialized T , so that T can have a faster convergence rate in pre-training. Targeting the above problems, bert2BERT first initializes the target model T with the parameters of the existing model S by the width-wise expansion ( D s  D t ) and depth-wise expansion ( L s  L t ). Through this expansion, the knowledge contained in the parameters of the source model is directly transferred to the target model. Then we further pre-train the initialized target model with a two-stage pre-training method. The overall workflow is illustrated in Section 4.5. Essentially, the width-wise expansion can be decomposed into expansions of parameter matrices (or vectors 3 ). As illustrated in Figure 3, the matrix expansion enlarges W  R d w in  d w out of S to U  R d u in  d u out of T by two kinds of operations: in-dimension and out-dimension expansion. In the following sections, we first introduce two strategies of width-wise expansion: function-preserving and advanced knowledge initialization. Then, we introduce the depth-wise expansion and detail the two-stage pre-training process. For the paper clarity, we introduce two index mapping functions: g in and g out , where g in ( i ) means the i -th in-dimension of U reuses the g in ( i ) -th in-dimension parameters of W , g out ( j ) means the j -th out-dimension of U reuses the g out ( j ) -th out-dimension parameters of W . Both our two methods are defined with these two mapping functions. W ( i,j ) means the parameter element, i and j refer to the i -th in-dimension index and j -th out-dimension index respectively. As shown in Figure 3, the i -th in-dimension parameters of W are the parameters of the i -th input neuron of W or the i -th column of W . Function preserving initialization (FPI) (Chen et al., 2016) aims to make the initialized target model have the same function as the source model, which means that given the same input, the initialized target model has the same output as the source model. In this paper, we extend FPI on a different architecture, Transformer-based pre-trained language model. We give an example in Figure 3 to illustrate 3 We omit the expansion of bias (vector) for simplicity. as follows: g in ( i ) = (cid:40) i i  [1 , d w in ] f ( { 1 , 2 , ..., d w in } ) i  ( d w in , d u in ] , (5) g out ( j ) = (cid:40) j j  [1 , d w out ] f ( { 1 , 2 , ..., d w out } ) j  ( d w out , d u out ] , (6) where f (  ) is uniform sampling. We denote the weight expansion as U = EXPN( W ; g in , g out ) , which includes in-dimension expansion (Eq. 7) and out-dimension expansion (Eq. 8): C g in ( i ) = d u in (cid:88) i  =1 I ( g in ( i  ) = g in ( i )) (cid:101) U ( i,  ) = 1 C g in ( i ) W ( g in ( i ) ,  ) , (7) U (  ,j ) = (cid:101) U (  ,g out ( j )) , (8) where I (  ) is an indicator function, and C g in ( i ) is the count of g in ( i ) in the values of g in (  ) , which is used to re-scale the original parameters to keep the function preserving property. Expansion for All Modules. We apply FPI for all modules of BERT via matrix expansion EXPN(  ) . Specifically, for the embedding matrix WE , we only conduct the out-dimension expansion: UE (  ,j ) = WE (  ,g e out ( j )) . (9) MHA module can be decomposed into multiple parallel self-attention heads and we conduct the head-wise expansion for this module, which means 2137 increasing the number of attention heads. The head-wise expansion is formulated as: UQ | K | V | O = EXPN( WQ | K | V | O ; g q | k | v | o in , g q | k | v | o out ) . (10) 2 and the out-dimension expansion for W Ql,i | W Kl,i | W Vl,i is: g q | k | v out ( j ) = (cid:40) j j  [1 , a s ] f ( { 1 , 2 , ..., a s } ) j  ( a s , a t ] , (11) where j is the head index and a s | t mean the head numbers of source model and target model respectively. Specifically, the head-wise expansion means that we reuse the head group parameters to construct the new matrices. The i -th head group in l -th layer contains W Ql,i | W Kl,i | W Vl,i | W Ol,i in Eq. The module has three constraints: { g e out = g q | k | v in ; g q | k | v out = g o in ; g q | k | v in = g o out }, with the first two constraints for hidden dimension consistency (Wen et al., 2018; Chen et al., 2021) and the third one for residual connection (Eq. 2). Similar to the MHA module, the mapping functions of FFN also have three constraints: { g o out = g 1in ; g 1out = g 2in ; g 1in = g 2out }. For the layer normalization, we take the layer normalization of FFN as an example, its expansion is formulated as: U LNj = W LNg 2out ( j ) . (13) Note that in layer normalization (Eq. 4), the mean  and variance  are calculated based on the hidden representations H . Thus, the expansion of this parameter inevitably induces a gap and prevents the target model from strictly following the function preserving principle. However, we empirically find that the gap is so small that it can hardly affect the initialization and convergence of the target model. Thus we ignore this discrepancy. We have validated the effectiveness of the adapted FPI in different settings in Table 1. The results show that the initialized model T achieves almost the same loss as S , demonstrating that FPI successfully retains the knowledge of the small model when performing parameter expansion. 4.3.2 Advanced Knowledge Initialization To further improve the convergence rate of the pretraining target model, we propose the advanced knowledge initialization (AKI), which expands new Method S (12 , 384) S (12 , 512) Original 1.89 1.67 Rand 10.40 10.42 DirectCopy 9.05 6.45 FPI 1.89 1.70 AKI 2.08 1.96 Table 1: The comparison of MLM losses between FPI and baselines. matrices based on not only the parameters of the same layer but also the parameters of the upper layer in the source model. The intuition is based on previous findings (Jawahar et al., 2019; Clark et al., 2019) that adjacent Transformer layers have similar functionality, which ensures that it will not damage the knowledge contained in the parameters of the current layer. Moreover, the knowledge that comes from adjacent layers can break the symmetry (Chen et al., 2016) appeared in FPI, which has been demonstrated beneficial. We give an illustrative example in Figure 4 and formulate AKI as: U l = EXPN( W l , W l +1 ; g l | l +1 in , g l out ) . (14) Specifically, we first do the in-dimension expansion for W l | l +1 . Here we take W l as an example: C g l in ( i ) = d u in (cid:88) i  =1 I ( g l in ( i  ) = g l in ( i )) (cid:101) U l ( i,  ) = 1 C g l in ( i ) W l ( g l in ( i ) ,  ) . (15) It is similar with Eq. 7. Then we stack the expanded matrices of (cid:101) U l and (cid:101) U l +1 to construct the final matrix: U l (  ,j ) = (cid:40) (cid:101) U l (  ,j ) j  [1 , d w out ] (cid:101) U l +1(  ,g l out ( j )) j  ( d w out , d u out ] . (16) We directly copy the expanded (cid:101) U l as the top part of the new matrix and place the sampled parameters from (cid:101) U l +1 on the bottom of the new matrix. We aggregate upper-layer information into a new matrix for two intuitions: (1) it breaks the FPI symmetry that hinders model convergence (Chen et al., 2138          in {1:1,2:2,:} 22   22 2  2   2  2  in+1 {1:1,2:2,:} 22   22  out {1:1,2:2,:} 2  2   Cheng 20211113 COPY  in  out  in  in  out  +1      +1   Figure 4: Overview of AKI. It first performs the in-dimension expansion on both the matrixes of current and upper layers. Then it uses the widened matrix of the current layer as the top part of the new matrix and samples the row of the widened matrix of the upper layer as the bottom part of the new matrix. 2016). For example, FPI makes the attention patterns in the same layer repeated, which is redundant and called symmetry; (2) upper-layer information can be used as similar but high-level knowledge to guide the model to converge faster. We display the attention patterns of the target model initialized by AKI in Appendix E and find that the target model can maintain the attention patterns of both current and upper layers very well. Expansion for All Modules. For embedding matrix, we only do the out-dimension expansion as Eq. 9 in the FPI. Both the modules of MHA and FFN do the matrix expansion by following the defined operation in Eq. 15 and Eq. 16. The constraints of mapping functions follow the setting of FPI.Empirically, we find that the AKI method outperforms FPI, while the performance is worse if we build a new matrix based on the matrix of the lower layer (or low-level knowledge). How to construct the optimal initialization for the target model with the parameters of different layers remains an open question and we leave it as future work. For more details, we give a clear illustration of the FPI and AKI process in Appendix F. 4.4 Depth-wise Expansion After the width-wise expansion, we obtain a widened model with the same width as the target model. To bridge the depth gap, we perform depthwise expansion to increase model depth to the depth of the target model. We illustrate this process in Algorithm 1 and the main idea is to iteratively stack the widened model until its depth is equal to the target model (Gong et al., 2019). Algorithm 2 Two-stage Pre-training Input: the initialized model T , large-scale unsupervised dataset D , the epoch number of sub-model training E b and the epoch number of whole training process E , the layer number l b . 1: Construct sub-models and these models have the layer numbers of { l b , 2  l b , . . . , L t }. 2: for e = 1  E b do 3: for batch in D do 4: T   sample one sub-model. 5: Perform forward and backward of T  . 6: Update only top l b layers of T  . 7: end for 8: end for 9: for e = E b  E do 10: for batch in D do 11: Perform forward and backward of T . 12: Update whole model T . 13: end for 14: end for Output: the pre-trained model T layers in a random manner to make the complete model converge at a low cost. These sub-models are built with bottom Transformer layers of the initialized target model and share one classification layer. At each optimization step, we randomly sample one sub-model and only update its top Transformer layers and the shared classification layer. (2) After the sub-structure training, we further perform the traditional full-model training. The details of our method are displayed in Algorithm 2. Pre-training Details. We use the English Wikipedia and Toronto Book Corpus (Zhu et al., 2015) as the pre-training data. The settings of pretraining are: peak learning rate of 1e-4, warmup 2139 Model FLOPs Ratio Loss SQuADv1.1 SST-2 MNLI MRPC CoLA QNLI QQP STS-B Avg. steps of 10k, training epochs of E =40, batch size of 512, sub-model training epochs of E b =5, layer number of l b =3. Unless otherwise noted, all methods including bert2BERT and baselines use the same pre-training settings for fair comparisons. In the settings of bert2BERT, the target model has a BERTBASE architecture of T (12 , 768) and the source model has an architecture of S (12 , 512) . Fine-tuning Details. For the evaluation, we use tasks from GLUE benchmark (Wang et al., 2019a) and SQuADv1.1 (Rajpurkar et al., 2016). We report F1 for SQuADv1.1, Matthews correlation coefficient (Mcc) for CoLA (Warstadt et al., 2019) and accuracy (Acc) for other tasks. For the GLUE tasks fine-tuning, we set the batch size to 32, choose the learning rate from {5e-6, 1e-5, 2e-5, 3e-5} and epochs from {4, 5, 10}. For the SQuADv1.1 fine-tuning, we set the batch size to 16, the learning rate to 3e-5, and the number of training epochs to 4. All results are the average of 3 runs on the dev set. Baselines. We first introduce a naive bert2BERT baseline named DirectCopy, which directly copies the small model to the target model and randomly initializes the unfilled parameters. StackBERT (Gong et al., 2019) and MSLT (Yang et al., 2020) are also included as the baselines. Both of them are trained in a progressive manner. Following the original setting, for the StackBERT, we first train the 3-layer BERT for 5 epochs, stack it twice into a 6-layer BERT and then train it for 7 epochs. In the final step, we stack the 6-layer model into BERTBASE and further train it with 28 epochs. For MSLT, we first perform 4-stage training. In each stage, we add the top 3 layers of the model already trained to the top of the model and then pre-train the new model by partially updating the top 3 layers. Each stage of the partial training process has 8 epochs. Finally, we further perform 20 full-model training epochs 4 to achieve the same loss as BERTBASE trained from scratch. The baselines are trained using the same optimizer, training steps, and warmup steps as the bert2BERT. We demonstrate the effectiveness of the proposed method on the SQuAD and GLUE benchmark. The results are shown in Table 2. We also represent the loss curves in Figure 1 and Appendix A. The results show that: (1) DirectCopy only saves 12.2% computational costs, which indicates this naive method of directly copying the trained parameters of the source model to the target model is not effective; (2) our proposed methods, FPI and AKI, achieve better performances than the baselines. Although AKI does not follow the function preserving, it has a bigger loss than FPI at the start of training, AKI achieves a faster convergence rate by using the advanced knowledge and breaking the symmetry; (3) by performing the two-stage pre-training on the target model initialized by AKI, we can save 45 . 2 % computational costs. Note that the total parameters of the source model are half of those of the target model (54M vs. 110M). The loss of bert2BERT in Figure 1 is high at the stage of sub-model training because it represents the average loss of all sub-models. We also compare the attention patterns of the target models initialized by DirectCopy, FPI, and AKI. The attention patterns and their discussions are displayed in Appendix E. 4 We have tried the same setting as the original paper with 8 epoch full-model running but it does not achieve the same loss with BERTBASE (1.511 vs. 1.437). bert2BERT with Smaller Source Model. We also evaluate bert2BERT on different settings, where the source model S (6, 512), S (8, 512), S (10, 512) are significantly smaller than the target model (35M | 42M | 48M vs. 110M). The results are shown in Table 3 and loss curves are displayed in Appendix B. We observe that DirectCopy for S (6, 512) achieves no efficiency improvement over the original pre-training, which indicates that the significant size gap between the source and target model greatly reduces the benefit of DirectCopy methods. Compared with DirectCopy, our proposed method reduces the computation cost by 23.3%, which again demonstrates the effectiveness of bert2BERT. The results show that the smaller the size gap between the source model and target model, the greater the cost savings of bert2BERT. We also note that it is more challenging to speed up the target model with a small source model S (6, 512). We encourage future work to explore to transfer the knowledge from smaller source models to improve the pre-training efficiency of the target model. Effect of Sub-model Training Epochs. Our training procedure includes two stages: sub-model training and full-model training. Here, we study the effect of the number of sub-model training epochs by performing bert2BERT on the different settings of E b ={0, 5, 10, 20}. The results are presented in Table 4 and the loss curves are displayed in Appendix C. We observe that our method achieves the best efficiency when the epoch number is set to 5, while a larger or smaller epoch number will bring a negative impact. Datasets. To demonstrate that our method is generic, following the BERT setting, we also use the English Wikipedia and Book Corpus in the GPT-training. For the evaluation, we use the datasets of WikiText-2, PTB, and WikiText103 and evaluate these models under the zero-shot setting without fine-tuning on the training set. Implementation Details. We use the architecture of { L =12, D =768} for the GPT target model, and pre-train it with the learning rate of 1e-4, training epochs of 20. For bert2BERT, we use the source model with an architecture of { L =12, D =512}, initialize the target model with AKI, and pre-train it by the full-model training ( E b =0). Results and Analysis. We compare the original pre-training method and bert2BERT, the results are shown in Table 5 and Appendix D. We observe that the proposed method saves 47% computation cost of GPT pre-training, exhibiting a similar trend to BERT pre-training. Although GPT and BERT have different architectures (e.g., post-LN and pre-LN (Xiong et al., 2020)) and are pretrained with different tasks, bert2BERT saves a significant amount of training cost on both these two models, which shows that the proposed method is generic and is effective for different kinds of PLMs. bert2BERT 2.6 ( 47 %  ) 132.1 47.9 53.0 Table 5: Experiments on GPT. We report the perplexity for these tasks. w/o FT means that the pre-trained model is directly evaluated on the test set without fine-tuning on the train set. Datasets. To demonstrate that our method can be used to train larger models, we use the Baidu Wikipedia, Sougou Wikipedia, and Zhihu to train the T5 model (Raffel et al., 2020). For the evaluation, we use the dataset of the original Chinese natural language inference task (OCNLI) (Hu et al., 2020). Implementation Details. Since the bert2BERT method is suitable for BERT and GPT, it can also be used for the T5 model, which consists of an encoder and a decoder. The target T5 model's architecture is { L e =12, L d =12, D =1024, A =16}, where 2141 L e and L d means the numbers of encoder and decoder Transformer layers respectively, D means the hidden size, A means the number of attention heads. We pre-train it with the learning rate of 1e-4, batch size of 1024. For bert2BERT, we use the source model with an architecture of { L e =12, L d =12, D =256, A =4}, initialize the target model with FPI, and pre-train it by the full-model training ( E b =0). Note that the scale gap between the source model and the target model is over 10 times (31M vs. 360M), which is a challenging setting. Results and Analysis. We compare the original pre-training method and bert2BERT method on the T5 model, the results are shown in Table 6. We observe that the proposed method saves at least 25% computation cost of T5 pre-training. It demonstrates the effectiveness of the method on larger models. This paper proposes an efficient pre-training method, bert2BERT, which reuses the parameters of the small trained model as the initialization parameters of the large model. We employ the proposed method in BERT and GPT under different settings of model sizes. The extensive results show that bert2BERT is generic to Transformer-based models and saves a significant amount of computation cost. Moreover, the detailed analysis shows that our techniques, function-preserving, advanced knowledge initialization, and two-stage pre-training, are all effective. In the future, we will apply bert2BERT on training super large-scale language models (e.g., use the 10B source model to train the 100B target model) and extends its scope to other PLMs such as ELECTRA and BART (Lewis et al., 2020). This work is supported in part by NSFC (Grant No. 61872215), and Shenzhen Science and Technology Program (Grant No. RCYX20200714114523079). We would like to thank Yifeng Liu, Binbin Deng, Ziliang Yang, Jiaxin Shi for their support of this work.
abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain abstain result result abstain objective result abstain other other other other other other other other abstain other other other other other other other objective abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method other abstain method abstain method method method abstain abstain other abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain method method method method abstain abstain method abstain abstain method method method method abstain method abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain objective objective abstain result abstain other other