Transformer-based pre-trained language models like BERT, though powerful in many tasks, are expensive in both memory and computation, due to their large number of parameters. Previous works show that some parameters in these models can be pruned away without severe accuracy drop. However, these redundant features contribute to a comprehensive understanding of the training data and removing them weakens the model's representation ability. In this paper, we propose GhostBERT, which generates more features with very cheap operations from the remaining features. In this way, GhostBERT has similar memory and computational cost as the pruned model, but enjoys much larger representation power. The proposed ghost module can also be applied to unpruned BERT models to enhance their performance with negligible additional parameters and computation. Empirical results on the GLUE benchmark on three backbone models (i.e., BERT, RoBERTa and ELECTRA) verify the efficacy of our proposed method. Recently, there is a surge of research interests in compressing the transformer-based pre-trained language models like BERT into smaller ones using various compression methods, i.e., knowledge distillation (Sanh et al., 2019; Sun et al., 2019; Jiao et al., 2020), pruning (Michel et al., 2019; Fan et al., 2019), low-rank approximation (Lan et al., 2020), weight-sharing (Lan et al., 2020), dynamic networks with adaptive depth and/or width (Liu et al., 2020; Hou et al., 2020; Xin et al., 2020; Zhou et al., 2020), and quantization (Shen et al., 2020; Fan et al., 2020; Zhang et al., 2020; Bai et al., 2021). Previous works show that there are some redundant features in the BERT model, and unimportant attention heads or neurons can be pruned away =   =   =   =   =   =   =  =   =   =   =  =  =   =   =  =   =   =   =   =   Figure 1: Average GLUE development accuracy versus #params and FLOPs with the (pruned) BERT and our GhostBERT. without severe accuracy degradation (Michel et al., 2019; Hou et al., 2020). However, for computer vision (CV) tasks, it is shown in (Han et al., 2020) that redundant features in convolutional neural networks also contribute positively to the performance, and using cheap linear operations to generate more ghost feature maps enhances the performance with few additional parameters. On the other hand, it is shown in (Voita et al., 2019; Kovaleva et al., 2019; Rogers et al., 2020) that many attention maps in pre-trained language models exhibit typical positional patterns, e.g., diagonal or vertical, which can be easily generated from other similar ones using operations like convolution. Based on the above two aspects, in this paper, we propose to use cheap ghost modules on top of the remaining important attention heads and neurons to generate more features, so as to compensate for the pruned ones. Considering that the convolution operation (1) encodes local context dependency, as a complement of the global self-attention in Transformer models (Wu et al., 2020); and (2) can generate some BERT features like positional attention maps from similar others, in this work, we propose to use the efficient 1-Dimensional Depthwise Separable Convolution (Wu et al., 2019) as the basic operation in the ghost module. To ensure the generated ghost features have similar scales or ghost features original features DWConv [CLS]the the mat [SEP] catsaton [CLS]the themat[SEP] cat sat on N d (pruned) MHA (pruned) FFN Input G-MHA Softmax G-FFN as the original ones, we use a softmax function to normalize the convolution kernel. Afterwards, we fine-tune the parameters in both the BERT backbone model and the added ghost modules. Note that the ghost modules are not necessarily applied to pruned models. They can also be directly applied to pre-trained language models for better performance while with negligible additional parameters and floating-point operations (FLOPs). Figure 1 summarizes the average accuracy versus parameter size and FLOPs on the GLUE benchmark, where adding ghost modules to both the unpruned ( m = 12 / 12 ) and pruned ( m < 1 ) BERT models perform better than the counterparts without ghost modules. More experiments on the GLUE benchmark show that with only 0 . 4% more parameters and 0 . 9% more FLOPs, the proposed ghost modules improve the average accuracy of BERT-base, RoBERTa-base and ELECTRA-small by 0 . 9 , 0 . 6 , 2 . 4 points, respectively. When applying ghost modules to small or pruned models, the resultant models outperform other BERT compression methods. In this section, we first introduce where to add ghost modules in a BERT model (Section 2.1), and then discuss the components and optimization details of the ghost module (Section 2.2). The BERT model is built with Transformer layers, each of which contains a Multi-Head Attention (MHA) layer and a Feed-Forward Network (FFN), as well as skip connections and layer normalizations. Hou et al. (2020) show that the computations for attention heads of MHA and neurons in the intermediate layer of FFN can be performed in parallel. Thus the BERT model can be compressed in a structured manner by pruning parameters associated with these heads and neurons (Hou et al., 2020). In this paper, after pruning the unimportant heads and neurons, we employ cheap ghost modules upon the remaining ones to generate more ghost features to compensate for the pruned ones. For simplicity of notation, we omit the bias terms in linear and convolution operations where applicable in the rest of this work. Following (Hou et al., 2020), we divide the computation of MHA into the computation of each attention head. Specifically, suppose the sequence length and hidden state size are n and d , respectively. Each transformer layer consists of NH attention heads. For input matrix X  R n  d , the h -th attention head computes its output as H h ( X ) = Softmax (1 /  d  XW Qh WK (cid:62) h X (cid:62) ) XW Vh WO (cid:62) h , where W Qh , W Kh , W Vh , W Oh  R d  d h with d h = d/N H are the projection matrices associated with it. In multi-head attention, NH heads are computed in parallel to get the final output: MHA ( X ) = NH (cid:88) h =1 H h ( X ) . Given a width multiplier m  1 , we keep M = (cid:98) N h m (cid:99) heads and use them to generate F ghost features. The f th ghost feature is generated by G f ( X )= Nonlinear (cid:32) M (cid:88) h =1 G f,h ( H h ( X )) (cid:33) . where G f,h is the proposed cheap ghost module which generates features from the h th attention head's representation to the f th ghost feature. ReLU is used as the nonlinearity function. Thus the computation of MHA in the GhostBERT is: Ghost-MHA ( X )= M (cid:88) h =1 H h ( X )+ F (cid:88) f =1 G f ( X ) . Similar to the attention heads in MHA, the computation of FFN can also be divided into computations for each neuron in the intermediate layer of FFN (Hou et al., 2020). With a slight abuse of notation, we still use X  R n  d as the input to FFN. Denote the number of neurons in the intermediate layer as d ff , the computation of FFN can be written as: FFN ( X ) = (cid:80) d ff i =1 GeLU (cid:16) XW 1: ,i (cid:17) W 2 i, : , where W 1 , W 2 are the weights in FFN. For simplicity, we also use width multiplier m for FFN as MHA, and divide these neurons into NH folds, where each fold contains d f = d ff /N H neurons. For the h -th fold, its output can be computed as H h ( X ) = GeLU (cid:0) XW 1 h (cid:1) W 2 h where W 1 h = W 1: , ( h  1) d f : hd f and W 2 h = W 2( h  1) d f : hd f , : are the parameters associated with it. In FFN, NH folds are computed in parallel to get the output: FFN ( X ) = NH (cid:88) h =1 H h ( X ) . For width multiplier m , we keep M folds of neurons and use ghost modules to generate F ghost features as in Equation (2). Thus the computation of FFN in the GhostBERT can be written as: Ghost-FFN ( X )= M (cid:88) h =1 H h ( X )+ F (cid:88) f =1 G f ( X ) . In the previous section, we discussed where we insert the ghost modules in the Transformer layer. In this section, we elaborate on the components and normalization of the ghost modules. Generally speaking, any function can be used as the ghost module G in Equation (2). Considering that (i) convolution operation can encode local context dependency, as a compensation for the global self-attention (Wu et al., 2020; Jiang et al., 2020); and (ii) features like diagonal or vertical attention maps (Kovaleva et al., 2019; Rogers et al., 2020) can be easily generated by convolving similar others, we consider using convolution as the basic operation in the ghost module. With a slight abuse of notation, here we still use X  R n  d as the input to the convolution, i.e., the output H h of h th head in MHA or h th fold of neurons in FFN. Denote O  R n  d as the output of the convolution in the ghost module. 1-Dimensional convolution (Conv1D) over the sequence direction encodes local dependency over contexts, and has shown remarkable performance for NLP tasks (Wu et al., 2019, 2020). To utilize the representation power of Conv1D without too much additional memory and computation, we choose 1-Dimensional Depthwise Separable Convolution (DWConv) (Wu et al., 2019) for the ghost module. Compared with Conv1D, DWConv performs a convolution independently over every channel, and reduces the number of parameters from d 2 k to dk (where k is the convolution kernel size). Denote the weight of the DWConv operation as W  R d  k . After applying DWConv, the output for the i th token and c th channel can be written as: O i,c = DWConv ( X : ,c , W c, : , i, c ) = k (cid:88) m =1 W c,m  X i (cid:100) k +12 (cid:101) + m,c . Since the parameters of the BERT backbone model and the ghost modules can have quite different scales and optimization behaviors, we use a softmax function to normalize each convolution kernel W c, : across the sequence dimension as Softmax( W c, : ) before convolution as Wu et al. (2019). By softmax normalization, the weights in one kernel are summed up to 1, ensuring that the convolved output has a similar scale as the input. Thus after applying the ghost module, the output for the i th token and c th channel can be written as:  O i,c = DWConv ( X : ,c , Softmax ( W c, : ) , i, c ) . To turn a pre-trained BERT model into a smaller-sized GhostBERT, we do the following three steps: Pruning. For a certain width multiplier m , we prune the attention heads in MHA and neurons in the intermediate layer of FFN from a pre-trained BERT-based model following (Hou et al., 2020). Distillation. Then we add ghost modules to the pruned model as in Section 2.1. Suppose there are L Transformer layers. We distill the knowledge from the embedding (i.e., the output of the embedding layer) E , hidden states M l after MHA and F l after FFN (where l = 1 , 2 ,    , L ) from the full-sized teacher model to E m , M ml , F ml of the student GhostBERT. Following (Jiao et al., 2020), we use the augmented data for distillation. Denote MSE as the mean squared error, the three loss terms are (cid:96) emb = MSE ( E m , E ) , (cid:96) mha = (cid:80) Ll =1 MSE ( M ml , M l ) , and (cid:96) ffn = (cid:80) Ll =1 MSE ( F ml , F l ) , respectively. Thus, the distillation loss function is: L distill = (cid:96) emb + (cid:96) mha + (cid:96) ffn . Note that instead of being applied to pruned models, the cheap ghost modules can also be directly applied to a pre-trained model for better performance while with negligible additional parameters and FLOPs. In this case, the training procedure contains only the distillation and fine-tuning steps. Empirically, to save memory and computation, we generate one ghost feature for each MHA or FFN (i.e., F = 1 in Equations (3) and (5)), and let all ghost modules G f,h share the same parameters with each other. As will be shown in Section 3, adding these simplified ghost modules already achieve clear performance gain empirically. In this section, we show the efficacy of the proposed method with (pruned) BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and ELECTRA (Clark et al., 2020) as backbone models. Experiments are performed on the GLUE benchmark (Wang et al., 2019), which consists of various natural language understanding tasks. More statistics about the GLUE datasets are in Appendix A.1. Following (Clark et al., 2020), we report Spearman correlation for STS-B, Matthews correlation for CoLA and accuracy for the other tasks. For MNLI, we report the results on the matched section. The convolution kernel size in the ghost module is set as 3 unless otherwise stated. The detailed hyperparameters for training the GhostBERT are in Appendix A.2. The model with the best development set performance is used for testing. For each method, we also report the number of parameters Model FLOPs(G) #params(M) MNLI QNLI QQP RTE SST-2 MRPC CoLA STS-B Avg. We compare our proposed method against the following methods: (i) baseline pre-trained language models: BERT-base (Devlin et al., 2019), RoBERTa-base (Liu et al., 2019) and ELECTRA-small (Clark et al., 2020); (ii) BERT compression methods: TinyBERT (Jiao et al., 2020), Con-vBERT (Jiang et al., 2020), and MobileBERT (Sun et al., 2020). The development set results of RoBERTa-base are from Hou et al. (2020). The test set results of ELECTRA, BERT-base and Con-vBERT are from Jiang et al. (2020). The others are from their original papers or repositories. Table 1 shows the GLUE development set results of the baseline pre-trained language models and our proposed method. When the cheap ghost modules are directly applied to these unpruned pre-trained models, better performances are achieved with only negligible additional parameters and FLOPs. Specifically, adding ghost modules to BERT-base, RoBERTa-base and ELECTRA-small increases the average development accuracy by 0 . 9 , 0 . 6 , 2 . 4 points with only 55 . 3 K more parameters, and 14 . 2 M more FLOPs. For the test set, the average performance gains are 0 . 8 , 1 . 1 , 2 . 4 points. Comparison with Baseline Models. From Table 1, when the ghost modules are applied to the pruned BERT (or RoBERTa) model with m < 1 , the proposed GhostBERT or GhostRoBERTa also achieves comparable performances as BERT-base or RoBERTa-base with fewer FLOPs. Specifically, GhostBERT ( m = 6/12) and GhostRoBERTa ( m = 9/12) perform similarly or even better than BERT-base and RoBERTa-base with only 50% and 75% FLOPs, respectively. In particular, when the compression ratio increases (i.e., m = 3/12, 1/12), we still achieve 99.6% performance (resp. 96.3%) with only 25% FLOPs (resp. 8%) of BERT-base model. Comparison with Other Compression Methods. Table 2 shows the comparison between the proposed method and other popular BERT compression methods. Under similar parameter sizes or FLOPs, the proposed GhostBERT performs comparably as the other BERT compression methods, while GhostRoBERTa often outperforms them. In particular, GhostELECTRA-small has over 1.5 points or higher accuracy gain than other similar-sized small models like ELECTRA-small, TinyBERT 4 and ConvBERT-small. In Table 3 and Figure 1, we also compare the pruned BERT with and without ghost modules. For fair comparison, for the pruned model without ghost module, we use the same training procedure as Section 2.3. As can be seen, adding the ghost modules achieves considerable improvement with negligible additional memory and computation. In this section, we perform ablation study in the (i) training procedure: including data augmentation (DA) and knowledge distillation (KD); (ii) ghost module: including convolution kernel size, softmax normalization over the convolution kernel and nonlinearity for each ghost feature in Equation (2). Training Procedure. Table 4 verifies the effectiveness of the Data Augmentation (DA) and Knowledge Distillation (KD) upon the GhostBERT model with width multiplier m  { 3 / 12 , 1 / 12 } . The GhostBERT incurs severe accuracy drop without DA and KD. with a drop of 3 . 5 and 6 . 4 points on average, for m = 3 / 12 and 1 / 12 , respectively. Ghost Module. Table 4 also shows the effectiveness of the softmax normalization over the convolution kernel and ReLU nonlinearity in Equation (2). As can be seen, dropping the softmax normalization or ReLU nonlinearity reduces the average accuracy by 0 . 8 and 1 . 6 points respectively for m = 3 / 12 , and 0 . 9 and 2 . 2 points respectively for m = 1 / 12 . Further, we explore whether the kernel size plays an important role in the DWConv in the ghost module. Figure 3 shows the results of GhostBERT with width multipliers m  { 3 / 12 , 1 / 12 } , with various convolution kernel sizes in DWConv. Average accuracy over five tasks is reported. Detailed results for each task can be found in Table 9 in Appendix B.1. As can be seen, the performance of GhostBERT increases first and then decreases gradually as the kernel size increases. For both width multipliers, kernel size 3 performs best and is used as the default kernel size in other experiments unless otherwise stated. In this section, we discuss about different choices of which type of convolution to use in the ghost module (Section 4.1), and where to posit the ghost modules in a BERT model (Section 4.2). Besides the DWConv in Section 2.2, in this section, we discuss more options for the convolution in the ghost module. We follow the notation in Section 2.2 and denote the input, output, kernel size of the convolution as X , W and k , respectively. 1-Dimensional Convolution. If the kernel convolves input over the sequence direction (abbre-viated as Conv1D S), the number of input and output channel is d , and the weight W has shape W  R d  d  k . After applying Conv1D S, the output for the i th token and c th channel is: O i,c = Conv1D S ( X , W c, : , : , i, c ) = d (cid:88) j =1 k (cid:88) m =1 W c,j,k  X i (cid:100) k +12 (cid:101) + m,j . If the kernel convolves input over the feature direction (abbreviated as Conv1D F), the number of input and output channel is n , and the weight has shape W  R n  n  k . After applying Conv1D F, the output for the i th token and c th channel is: O i,c = Conv1D F ( X , W i, : , : , i, c ) = n (cid:88) j =1 k (cid:88) m =1 W i,j,m  X j,c (cid:100) k +12 (cid:101) + m . 2-Dimensional Convolution (Conv2D). For Conv2D, the number of input and output channels are both 1, and thus the weight W has shape W  R 1  1  k  k . After applying Conv2D, the output for the i th token and c th channel is: O i,c = Conv2D ( X , W , i, c ) = k (cid:88) w =1 k (cid:88) h =1 W : , : ,h,w  X i (cid:100) k +12 (cid:101) + h,c (cid:100) k +12 (cid:101) + w . Table 5 shows the comparison of using different convolutions for the ghost module. For 1-Dimensional convolution, Conv1D S performs better Conv1D F. This may because that convolving over the sequence urges the model to learn the dependencies among tokens. Though 2-Dimensional convolution (Conv2D) is quite successful in CV tasks, it performs much worse than Conv1D S here. This may because the two dimensions of feature maps in CV tasks encode similar information, while those of hidden states in Transformers encode quite different information (i.e., feature and sequence). Thus Conv2D results in worse performance than Conv1D S, though much fewer parameters and FLOPs are required. On the other hand, DWConv achieves comparable performance as Conv1D S, while being much more efficient in terms of number of parameters and FLOPs, by performing the convolution independently over every feature dimension. In this section, we explore more possible positions of adding the ghost module. For MHA, besides adding ghost module after the projection layer ( After O in Figure 4(c)) as in Section 2.1.1, we can also add it right after calculating the attention score ( After QK in Figure 4(a)), or after multiplying the attention score and the value layer ( After V in Figure 4(b)). For FFN, besides adding the ghost module after the second linear layer ( After FFN2 in Figure 4(e)) as in Section 2.1.1, we can also add it after the intermediate layer ( After FFN1 in Figure 4(d)). Note that we use Conv2D as the ghost module for After QK because the attention map encodes attention probabilities in both dimensions. For After QK and After V , to match the dimension of other parameters, the number of input and output channels are M and NH  M , respectively. Table 6 shows the results of adding one ghost module to the same position for each Transformer layer. As can be seen, adding ghost module upon the attention maps ( After QK ) performs best. However, since the parameters in the value and projection layer of MHA are left unpruned, After QK has much more parameters and FLOPs than the other positions. Adding ghost modules to the other four positions has similar average accuracy. Thus in this work, for MHA, we choose the most memoryand computation-efficient strategy After O . Similarly, for FFN, we also add ghost modules to the final output ( After FFN2 ). From Table 6, our way of adding ghost modules has comparable performance as After QK , while being much more efficient in parameter size and FLOPs. Pruning removes unimportant connections or neurons in the network. Compared with pruning connections (Yu et al., 2019; Gordon et al., 2020; Sanh Linear Linear Linear Input Mat Mul Scale & Softmax Ghost Mat Mul NonLinear Linear Input Ghost NonLinear Linear Input Ghost Linear Linear Linear Input Mat Mul Scale & Softmax Ghost Mat Mul Linear Linear Linear Input Mat Mul Scale & Softmax Ghost Mat Mul The first group of research works replaces the self-attention mechanism or feed-forward networks with simpler and more efficient convolution operations, while maintaining comparable results. Wu et al. (2019) introduce the token-based dynamic depth-wise convolution to compute the importance of context elements, and achieve better results in various NLP tasks. Iandola et al. (2020) replace all the feed-forward networks with grouped convolution. AdaBERT (Chen et al., 2020) uses differentiable neural architecture to search for more efficient convolution-based NLP models. et al., 2020), structured pruning prunes away a group of parameters without changing the model topology and is more favored for hardware and real inference speedup. In the width direction, Michel et al. (2019); Voita et al. (2019) retain the performance after pruning a large percentage of attention heads in a structured manner. Besides attention heads, McCarley et al. (2019) also prune the neurons and the embeddings. In the depth direction, pruning Transformer layers is proposed in LayerDrop (Fan et al., 2019) via structured dropout. Efficient choice of Transformer layers at inference via early exit are also proposed in (Liu et al., 2020; Xin et al., 2020; Zhou et al., 2020). Hou et al. (2020) perform structured pruning in both width and depth directions. The importance of attention heads and neurons in the intermediate layer of Feed-forward network is measured by their impact on the loss, and the least important heads and neurons are pruned away. Various methods have been proposed to use linear or convolution operations to enhance the representation of the Transformer layers. The second group uses linear or convolutional module along with the self-attention mechanism for more powerful representation. The new module can be incorporated though serial connection to the original self-attention mechanism (Mehta et al., 2020), or be used in parallel with the original self-attention mechanism (Wu et al., 2020; Jiang et al., 2020) to capture both local and global context dependency. Serial and parallel connections of these linear or convolution operations to Transformer layers are also extended to multi-task (Houlsby et al., 2019; Stickland and Murray, 2019) and multilingual tasks (Pfeiffer et al., 2020). Note that the proposed ghost modules are orthogonal to the above methods in that these modules are used to generate more features for the Transformer models and can be easily integrated into existing methods to boost their performance. In this paper, we propose GhostBERT to generate more features in pre-trained model with cheap operations. We use the softmax-normalized 1-Dimensional Convolutions as ghost modules and add them to the output of the MHA and FFN of each Transformer layer. Empirical results on BERT, RoBERTa and ELECTRA demonstrate that adding the proposed ghost modules enhances the representation power and boosts the performance of the original model by supplying more features. We thank MindSpore for the partial support of this work, which is a new deep learning computing framework. Given the superior performance of Huawei Ascend AI Processor and MindSpore computing framework, our code will be released based on MindSpore at ( https: //gitee.com/mindspore/mindspore/tree/ master/model_zoo/research/nlp/ghostbert ).
abstain abstain abstain objective abstain abstain objective other result abstain abstain abstain objective other abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain abstain other abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain method method abstain abstain abstain method method abstain abstain method abstain abstain abstain method abstain abstain method abstain method abstain method abstain method method abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method method method other other abstain abstain abstain abstain other other other other other other other abstain abstain abstain abstain abstain objective method abstain other other