Transformer-based pre-trained models, such as BERT, have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, deploying these models can be prohibitively costly, as the standard self-attention mechanism of the Transformer suffers from quadratic computational cost in the input sequence length. To confront this, we propose FCA, a fineand coarse-granularity hybrid self-attention that reduces the computation cost through progressively shortening the computational sequence length in self-attention. Specifically, FCA conducts an attention-based scoring strategy to determine the informativeness of tokens at each layer. Then, the informative tokens serve as the fine-granularity computing units in self-attention and the uninformative tokens are replaced with one or several clusters as the coarse-granularity computing units in self-attention. Experiments on GLUE and RACE datasets show that BERT with FCA achieves 2x reduction in FLOPs over original BERT with <1% loss in accuracy. We show that FCA offers significantly better trade-off between accuracy and FLOPs compared to prior methods 1 . Transformer-based large pre-trained language models with BERT (Devlin et al., 2018) as a typical model routinely achieve state-of-the-art results on a number of natural language processing tasks (Yang et al., 2019; Liu et al., 2019; Clark et al., 2020), such as sentence classification (Wang et al., 2018), question answering (Rajpurkar et al., 2016, 2018), and information extraction (Li et al., 2020b). practicality, especially in the case of limited industry time and resources, such as Mobile Phone and AIoT. In addition, the excessive energy consumption and environmental impact caused by the computation of these models also raise the widespread concern (Strubell et al., 2019; Schwartz et al., 2020). To improve the efficiency of BERT, the mainstream techniques are knowledge distillation (Hin-ton et al., 2015) and pruning. Knowledge distillation aims to transfer the knowledge" from a large teacher model to a lightweight student model. The student model is then used during inference, such as DistilBERT (Sanh et al., 2019). Pruning technique includes: (1) structured methods that prune structured blocks of weights or even complete architectural components in BERT, for example encoder layers (Zhang and He, 2020), (2) unstructured methods that dynamically drop redundant units, for example, attention head (Voita et al., 2019) and attention tokens (Goyal et al., 2020). However, both types of methods encounter challenges. For the former, a great distillation effect often requires an additional large teacher model and very complicated training steps (Jiao et al., 2019; Hou et al., 2020). For the latter, pruning methods discard some computing units, which inevitably causes information loss. In contrast to the prior approaches, we propose a self-motivated and information-retained technique, namely FCA, a fineand coarse-granularity hybrid self-attention that reduces the cost of BERT through progressively shortening the computational sequence length in self-attention. Specifically, FCA first evolves an attention-based scoring strategy to assign each token with the informativeness. Through analyzing the informativeness distribution at each layer, we conclude that maintaining full-length token-level representations is progressive redundant along with layers, especially for the classification tasks that only require single-vector repre-4811 sentations of sequences. Consequently, the tokens are divided into informative tokens and uninformative tokens according to their informativeness. Then, they are updated through different computation paths. The informative tokens carry most of the learned features and remain unchanged as the fine-grained computing units in self-attention. The uninformative tokens may not be as important as informative ones but we will not completely discard them to avoid information loss. Instead, We replace them with more efficient computing units to save memory consumption. Experiments on the standard GLUE benchmark show that FCA accelerates BERT inference speed and maintains high accuracy as well. Our contributions are summarized as follows:  We analyze the progressive redundancies in maintaining full-length token-level representations for the classification tasks.  We propose a fineand coarse-granularity hybrid self-attention, which is able to reduce the cost of BERT and maintain high accuracy.  Experiments on the standard GLUE benchmark show that the FCA-based BERT achieves 2x reduction in FLOPs over the standard BERT with < 1% loss in accuracy. 2 Related work There has been much prior literature on improving the efficiency of Transformers. The most common technologies include: Knowledge distillation refers to training a smaller student model using outputs from various intermediate representations of larger pre-trained teacher models. In the BERT model, there are multiple representations that the student can learn from, such as the logits in the final layer, the outputs of the encoder units, and the attention maps. The distillation on output logits is most commonly used to train smaller BERT models (Sanh et al., 2019; Sun et al., 2019; Jiao et al., 2019; Sun et al., 2020). The output tensors of encoder units contain meaningful semantic and contextual relationships between input tokens. Some work creates a smaller model by learning from the outputs of teacher's encoder (Jiao et al., 2019; Sun et al., 2020; Li et al., 2020a). Attention map refers to the softmax distribution output of the self-attention layers and indicates the contextual dependence between the input tokens. A common practice of distillation on attention maps is to directly minimize the difference between the self-attention outputs of the teacher and the student (Jiao et al., 2019; Sun et al., 2020; Mao et al., 2020). This line of work is orthogonal to our approach and our proposed FCA can be applied to the distillate models to further accelerate their inference speed. Pruning refers to identifying and removing less important weights or computation units. Pruning methods for BERT broadly fall into two categories. Unstructured pruning methods prune individual weights by comparing their absolute values or gradients with a pre-defined threshold (Mao et al., 2020; Gordon et al., 2020; Chen et al., 2020). The weights lower than the threshold are set to zero. Unlike unstructured pruning, structured pruning aims to prune structured blocks of weights or even complete architectural components in the BERT model. Voita et al. (2019) pruned attention heads using a method based on stochastic gates and a differentiable relaxation of the L0 penalty. Fan et al. (2019) randomly dropped Transformer layers to sample small sub-networks from the larger model during training which are selected as the inference models. Goyal et al. (2020) progressively reduced sequence length by pruning word-vectors based on the attention values. This work is partly similar to the fine-grained computing units in our proposed FCA. However they ignored the coarse-grained units that may cause information loss. In addition, there are some engineering techniques to speed up the inference speed, such as Mixed Precision (Micikevicius et al., 2017) and Quantization (Zafrir et al., 2019; Fan et al., 2020). Using half-precision or mixed-precision representations of floating points is popular in deep learning to accelerate training and inference speed. Quantization refers to reducing the number of unique values required to represent the model weights, which in turn allows to represent them using fewer bits. 3 Preliminary BERT (Devlin et al., 2018) is a Transformer-based language representation model, which can be fine-tuned for many downstream NLP tasks, including sequence-level and token-level classification. The Transformer architecture (Vaswani et al., 2017) is a highly modularized neural network, where each Transformer layer consists of two submodules, namely the multi-head self-attention sub-4812 Standard Deviation Variance Layer1 Layer2 Layer3 Layer6 Layer12 Figure 1: The first sub-figure is the normalized variance and standard deviation of informativeness with respect to BERT-base layers from 1 to 12. The last five sub-figures are the informativeness distributions on some layers. layer (MHA) and the position-wise feed-forward network sub-layer (FFN). Both sub-modules are wrapped by a residual connection and layer normalization. MHA . The self-attention mechanism allows the model to identify complex dependencies between the elements of each input sequence. It can be formulated as querying a dictionary with key-value pairs. Formally, MHA ( Q, K, V ) = Concat ( head 1 , ..., head h ) WO (1) where Q, K , and V represent query, key, and value. h is the number of heads. Each head is defined as: head t = Attention ( QW Qt , KW Kt , V W Vt ) = softmax ( QW Qt ( KW Kt ) T  d K ) (cid:124) (cid:123)(cid:122) (cid:125) AV W Vt (2) where W Qt  R d h  d Q , W Kt  R d h  d K , W Vt  R d h  d V , WO  R hd V  d h are learned parameters. d K , d Q , and d V are dimensions of the hidden vectors. The main cost of MHA layer is the calculation of attention mapping matrix A  R n  n in Eq. 2 which is O ( n 2 ) in time and space complexity. This quadratic dependency on the sequence length has become a bottleneck for Transformers. FFN . The self-attention sub-layer in each of the layers is followed by a fully connected position-wise feed-forward network, which consists of two linear transformations with a GeLU (Hendrycks and Gimpel, 2016) activation in between. Given a vector x i in [ x 1 , ..., x n ] outputted by MHA sublayer, FFN is defined as: FFN ( x i ) = GeLU ( x i W 1 + b 1 ) W 2 + b 2 , (3) where W 1 , W 2 , b 1 , b 2 are learned parameters. Previous research (Ganesh et al., 2021) has shown that in addition to MHA sub-layer, FFN sub-layer also consumes large memory in terms of model size and FLOPs. As a result, if we reduce the computational sequence length of MHA, the input and the consumption of FFN sub-layer will become less accordingly. 4 Methodologies To shorten the computational sequence length of self-attention, our core motivation is to divide tokens into informative and uninformative ones and replace the uninformative tokens with more efficient units. This section introduces each module of our model in detail. 4.1 Scoring Strategy Our strategy of scoring the informativeness of tokens is based on the self-attention map. Concretely, taking a single token vector x i as an example, its attention head x ( t ) i is updated by: x ( t ) i = (cid:80) nj =1 a i,j x ( t ) j (Eq. 2). a i,j is an element in attention map A . Therefore, a i,j represents the information contribution from token vector x j to x i over head t . Intuitively, we define the informativeness of a token by accumulating along the columns of 4813 Granularity Hybrid Layer  Granularity Hybrid Layer Multi-head Self-Attention Multi-head Self-Attention Layer1 Layer2 ! "  0.21 0.02 0.01 0.04 0.19 0.11 0.32 ! The overall informativeness of x j is defined as the average over the heads: We next analyze some properties of defined informativeness in BERT-base. The first sub-figure in Figure 1 displays the normalized variance and standard deviation of informativeness of layers from 1 to 12 on RTE (classification dataset), which supports the phenomenon that the informativeness distributions at the bottom layers are relatively uniform and the top layers are volatile. The last five sub-figures further present the informativeness distributions of some BERT-base layers, where the first token is [CLS] and its representations are used for the final prediction. We can see that as the layers deepen, the informativeness is progressively concentrated on two tokens. This means that maintaining full-length token-level representations for the classification tasks may be redundant. A straightforward approach for reducing the sequence length of self-attention is to maintain the informative tokens and prune the rest. We argue that this approach is effortless but encounters the risk of information loss, especially for lower layers. Instead of pruning, we propose to process the uninformative tokens with more efficient units. Figure 2 shows the architecture of FCA layer, which inserts a granularity hybrid sub-layer after MHA. At each layer, it first divides tokens into informative and uninformative ones based on their assigned informativeness. The CLS token is always divided into informative part as it is used to derive the final prediction. Let x ( l ) cls  X ( l ) be the sequence of token vectors input to l -th layer, where X ( l ) = [ x ( l ) 1 , ..., x ( l ) n ] and n is the sequence length of X ( l ) . We gather the token vectors from X ( l ) with the topk informativeness to form the informative sequence X ( l ) in and the rest vectors to form the uninformative sequence X ( l ) un , where X ( l ) in  R k  d h and X ( l ) un  R ( n k )  d h . The length of the uninformative sequence is reduced by performing certain type of aggregating operations along the sequence dimension, such as 4814 average pooling : X  ( l ) un = Pooling ( X ( l ) un ) (6) or weighted average pooling with informativeness as weights:  x ( l ) un = softmax ( I ( x ( l ) un )) X  ( l ) un = Pooling ( X ( l ) un ) (7) where x ( l ) un is the token vector in X ( l ) un . The aggregated sequence X  ( l ) un  R k   d h and k  is a fixed parameter. After hybrid layer, token sequence is updated to [ x ( l ) cls  X ( l ) in  X  ( l ) un ] and sequence length is shortened by n k k  . Therefore, in addition to the following layers, the computation cost of FFN in l -th layer is reduced as well. It should be noted that the relative position of uninformative tokens should be preserved, which contains their contextual features to a certain extent and they can be captured by aggregating operations. The parameter k is learnable and progressively shortened. Inspired by Goyal et al. (2020), we train n learnable parameters to determine the configuration of k , denoted R = [ r 1 , ..., r n ] . The parameters are constrained to be in the range, i.e., r i  [0; 1] and added after MHA sub-layer. Given a token vector x i output by MHA, it is modified by: x i  r pos ( x i ) x i (8) where pos ( x i ) is the sorted position of x i over informativeness. Intuitively, the parameter r i represents the extent to which the informativeness of the token at i -th position is retained. Then, for the l -th layer, we obtain the configuration of k l from the sum of the above parameters, i.e., k l = ceil ( sum ( l ; R )) s.t. k l +1  k l (9) 5 Loss Function Let  be the parameters of the baseline BERT model and L (  ) be cross entropy loss or mean-squared error as defined in the original task. We adopt the multi-task learning idea to jointly minimize the loss in accuracy and total sequence length over all layers. where L is the number of layers. L ( , R ) controls the accuracy and sum ( l ; R ) controls the sequence length of l -th layer. The hyper-parameter  tunes the trade-off. The training schema of our model involves three stages, which are given in Algorithm 1. Our experiments are mainly conducted on GLUE (General Language Understanding Evaluation) 2 (Wang et al., 2018) and RACE (Lai et al., 2017) datasets. GLUE benchmark covers four tasks: Linguistic Acceptability, Sentiment Classification, Natural Language Inference, and Paraphrase Similarity Matching. RACE is the Machine Reading Comprehension dataset. For experiments on RACE, we denote the input passage as P , the question as q , and the four answers as { a 1 , a 2 , a 3 , a 4 } . We concatenate passage, question and each answer as a input sequence [ CLS ] P [ SEP ] q [ SEP ] a i [ SEP ] , where [ CLS ] and [ SEP ] are the special tokens used in the original BERT. The representation of [ CLS ] is treated as the single logit value for each a i . Then, a softmax layer is placed on top of these four logits to obtain 2 https://gluebenchmark.com/ 4815 the normalized probability of each answer, which is used to compute the cross-entropy loss. The input length of BERT is set to 512 by default. However, the instances in these datasets are relatively short, rarely reaching 512. If we keep the default length settings, most of the input tokens are [PAD] tokens. In this way, our model can easily save computational resources by discriminating [PAD] tokens as the uninformative ones, which is meaningless. To avoid this, we constrained the length of the datasets. The statistic information of the datasets is summarized in Table 1. For accuracy evaluation, we adopt Matthew's Correlation for CoLA, F1-score for QQP, and Accuracy for the rest datasets. For efficiency evaluation, we use the number of floating operations (FLOPs) to measure the inference efficiency, as it is agnostic to the choice of the underlying hardware. We compare our model with both distillation and pruning methods. Distillation methods contain four models DistilBERT (Sanh et al., 2019), BERT-PKD (Sun et al., 2019), Tiny-BERT (Jiao et al., 2019), and Mobile-BERT (Sun et al., 2020). All four models are distillation from BERT-base and have the same structure (6 transformer layers, 12 attention heads, dimension of the hidden vectors is 768). Pruning methods contain FLOP (Wang et al., 2020), SNIP (Lin et al., 2020), and PoWER-BERT (Goyal et al., 2020). PoWER-BERT (Goyal et al., 2020) is the state-of-the-art pruning method which reduces sequence length by eliminating word-vectors. To make fair comparisons, we set the length of our informative tokens at each layer same to the sequence length of PoWER-BERT. We deploy BERT-base as the standard model in which transformer layers L = 12 , hidden size d h = 512 , and number of heads h = 12 . All models are trained with 3 epochs. The batch size is selected in list 16,32,64. The model is optimized using Adam (Kingma and Ba, 2014) with learning rate in range [2e-5,6e-5] for the BERT parameters  , [1e-3,3e-3] for R . Hyper-parameter  that controls the trade-off between accuracy and FLOPs is set in range [1e-3,7e-3]. We conducted experiments with a V100 GPU. The FLOPs for our model and the baselines were calculated with Tensorflow and batch size=1. Table 3 and Table 2 display the accuracy and inference FLOPs of each model on GLUE benchmark respectively. As the FLOPs of PoWER-BERT is almost the same as that of FCA-BERT and the number of coarse units has little affect on FLOPs, Table 2 only lists the FLOPs of FCA-BERT. Comparison to BERT . The results demonstrate the high-efficiency of our model, which almost has no performance gap with BERT-base (<%1 accuracy loss) while reduces the inference FLOPs by half on majority datasets. Table 4 presents the sequence length of FCA at each layer, which illustrates substantial reduction of computation length for standard BERT. For example, the input sequence length for the dataset QQP is 128. Hence, standard BERT needs to process 128*12=1536 tokens over the twelve layers. In contrast, FCA only tackles [85, 78, 73, 69, 61, 57, 54, 52, 46, 41, 35, 35] summing to 686 tokens. Consequently, the computational load of self-attention and the feed forward network is economized significantly. Among our models, the weighted average pooling operation raises the better performance than the average pooling operation. The number of coarse units contributes the model accuracy for both two operations, especially for pooling operation. This is reasonable as when the number of coarse units increases, the information stored in each FCA gradually approaches the standard BERT. But overmuch coarse units grow FLOPs. Therefore, it is necessary to balance impact on FLOPS and performance brought by the coarse units. Comparison to Baselines . We first compare our model to Distil-BERT. Our models dramatically outperform Distil-BERT in accuracy by a margin of at least 3 average score. As mentioned before, the line of distillation framework is orthogonal to our proposed method. We further investigate whether FCA is compatible with distillation models. Table 5 shows the results of Distil-BERT with FCA-Pool 5 , which verify that FCA could further accelerate the inference speed on the basis of the distillation model with <1% loss in accuracy. As for the SOTA distillation models, Tiny-BERT and Mobile-BERT, our models still outperform them on average performance. Combined with the results of Table 2 where our models have slightly fewer 4816 Dataset CoLA RTE QQP SST-2 MNLI-m QNLI RACE Avg. inference FLOPs than the distillation methods, it can be proved that FCA has better accuracy and computational efficiency than them. We next compare our model to the SOTA pruning model PoWER-BERT. Their acceleration effects are comparable and we focus on comparing their accuracy. The results on Table 3 show that our models achieve better accuracy than PoWER-BERT on all datasets. This is because PoWER-BERT discards the computing units, which inevitably causes information loss. Instead of pruning, FCA layer stockpiles the information of uninformative tokens in a coarse fashion (aggregating operations). Moreover, we noticed that coarse units are not always classified as uninformative. In other words, they sometimes participate in the calculation of self-attention as informative tokens. This shows the total informativeness contained in uninformative tokens can not be directly negligible and can be automatically learned by self-attention. In order to visually demonstrate the advantages of our model, Figure 3 draws curves of trade-off between accuracy and efficiency on three datasets. The results of FCA-BERT and PoWER-BERT are obtained by tuning the hyper-parameter  . For DistilBERT, the points correspond to the distillation version with 4 and 6 Transformer layers. It can be seen that with the decrease of FLOPs, (1) PoWER-BERT and our model outperform DistilBERT by a large margin; (2) our model exhibits the superiority over all the prior methods consistently; (3) more importantly, the advantage of our model over PoWER-BERT gradually becomes apparent. This is because PoWER-BERT prunes plenty of computation units to save FLOPs, which results in the dilemma of information loss. In contrast, our model preserves all information to a certain extent. Extensions to Other PLMs . To explore the generalization capabilities of FCA, we extend FCA to other pre-trained language models (PLMs), such as distil-BERT, BERT-large, and ELECTRA-base (Clark et al., 2020). The test results are displayed in Table 5, which proves that FCA is applicable to a variety of models, regardless of model size and variety. In this section, we explore that can we not differentiate between tokens and perform the average pooling on all tokens to reduce the computation cost. To make fair comparisons, we set the length of pooled sequence at each layer equal to the FCA-BERT-Pool 5 . The results show that pooling all tokens decreases the model accuracy from 75.0 to 73.8. This is because the pooling operation weakens the semantic features learned by the informative tokens, which are often decisive for the final prediction. On the contrary, our model does not conduct pooling on informative tokens and instead delegates the burden of saving computational overhead to uninformative tokens. And this does not cause serious damage to the representative features learned by the model. In this section, we further investigate the extent to which these compressed models can retain the essential information of the original BERT. Concretely, we adopt the Euclidean distance of the CLS representation between BERT and the compressed models as the evaluation metric, which is proportional to the information loss caused by model compression, formally: Distance(A,B) = M (cid:88) k =1 (cid:118)(cid:117)(cid:117)(cid:116) d h (cid:88) i =1 ( A clsi,k  B clsi,k ) where M is the number of the instances in corresponding dataset. Table 4 shows the distance of baselines and our models with standard BERT. Combining the results in Table 3, it can be found that the distance is consistent with the test accuracy. Large distance leads to low accuracy and vice versa. This provides an inspiration, that is, we can add a distance regulation term to the objective function to forcibly shorten the distance between the compression model and the original BERT, i.e., Loss  ,R = L ( , R )+  L (cid:88) l =1 l  sum ( l ; R )+ Distance (  ) However, the experimental results show that the accuracy has not been significantly improved. This may be because the information learned by the compressed model has reached the limit of approaching the BERT, and the regulation term can not further improve the potential of the compressed model. Our proposed FCA is dedicated to the classification tasks that only require single-vector representations, and it can not be directly applied to the tasks of requiring to maintain the full input sequence in the output layer, such as NER and extractive MRC. On these tasks, we need to make some modifications of only performing FCA operation over K and V in self-attention and maintaining the full length of Q . The Eq. 2 is modified to: head t = Attention ( QW Qt , FCA ( K ) W Kt , FCA ( V ) W Vt ) (11) We also attempted to maintain the full length of K and V and shorten Q , but the experimental results are unsatisfactory. In this paper, we propose FCA, a fineand coarse-granularity hybrid self-attention that reduces the computation cost through progressively shortening the computational sequence length in self-attention. Experiments on GLUE and RACE datasets show that BERT with FCA achieves 2x reduction in FLOPs over original BERT with <1% loss in accuracy. Meanwhile, FCA offers significantly better trade-off between accuracy and FLOPs compared to prior methods. We would like to thank three anonymous reviewers for their useful feedback. This work is supported by the National Key Research and Development Program of China under Grant No. 2020AAA0108600.
abstain abstain objective abstain abstain abstain result other abstain abstain abstain other abstain method abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain abstain method abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain other abstain abstain abstain method abstain result objective abstain abstain other other