Pre-trained language models have shown stellar performance in various downstream tasks. But, this usually comes at the cost of high latency and computation, hindering their usage in resource-limited settings. In this work, we propose a novel approach for reducing the computational cost of BERT with minimal loss in downstream performance. Our method dynamically eliminates less contributing tokens through layers, resulting in shorter lengths and consequently lower computational cost. To determine the importance of each token representation, we train a Contribution Predictor for each layer using a gradient-based saliency method. Our experiments on several diverse classification tasks show speedups up to 22x during inference time without much sacrifice in performance. We also validate the quality of the selected tokens in our method using human annotations in the ERASER benchmark. In comparison to other widely used strategies for selecting important tokens, such as saliency and attention , our proposed method has a significantly lower false positive rate in generating rationales. Our code is freely available at https://github.com/amodaresi/ AdapLeR . While large-scale pre-trained language models exhibit remarkable performances on various NLP benchmarks, their excessive computational costs and high inference latency have limited their usage in resource-limited settings. In this regard, there have been various attempts at improving the efficiency of BERT-based models (Devlin et al., 2019), including knowledge distilation (Hinton et al., 2015; Sanh et al., 2019; Sun et al., 2019, 2020; Jiao et al., 2020), quantization (Gong et al., 2014; Shen et al., 2020; Tambe et al., 2021), weight  Equal Contribution. pruning (Han et al., 2016; He et al., 2017; Michel et al., 2019; Sanh et al., 2020), and progressive module replacing (Xu et al., 2020). Despite providing significant reduction in model size, these techniques are generally static at inference time, i.e., they dedicate the same amount of computation to all inputs, irrespective of their difficulty. A number of techniques have been also proposed in order to make efficiency enhancement sensitive to inputs. Early exit mechanism (Schwartz et al., 2020b; Liao et al., 2021; Xin et al., 2020; Liu et al., 2020; Xin et al., 2021; Sun et al., 2021; Eyzaguirre et al., 2021) is a commonly used method in which each layer in the model is coupled with an intermediate classifier to predict the target label. At inference, a halting condition is used to determine whether the model allows an example to exit without passing through all layers. Various halting conditions have been proposed, including Shannon's entropy (Xin et al., 2020; Liu et al., 2020), softmax outputs with temperature calibration (Schwartz et al., 2020b), trained confidence predictors (Xin et al., 2021), or the number of agreements between predictions of intermediate classifiers (Zhou et al., 2020). Most of these input-adaptive techniques compress the model from the depth perspective (i.e., reducing the number of involved encoder layers). However, one can view compression from the width perspective (Goyal et al., 2020; Ye et al., 2021), i.e., reducing the length of hidden states. (Ethayarajh, 2019; Klafka and Ettinger, 2020). This is particularly promising as recent analytical studies showed that there are redundant encoded information in token representations (Klafka and Ettinger, 2020; Ethayarajh, 2019). Among these redundancies, some tokens carry more task-specific information than others (Mohebbi et al., 2021), suggesting that only these tokens could be considered through the model. Moreover, in contrast to layer-wise pruning, token-level pruning does not 1 come at the cost of reducing model's capacity in complex reasoning (Sanh et al., 2019; Sun et al., 2019). PoWER-BERT (Goyal et al., 2020) is one of the first such techniques which reduces inference time by eliminating redundant token representations through layers based on self-attention weights. Several studies have followed (Kim and Cho, 2021; Wang et al., 2021); However, they usually optimize a single token elimination configuration across the entire dataset, resulting in a static model. In addition, their token selection strategies are based on attention weights which can result in a suboptimal solution (Ye et al., 2021). In this work, we introduce Adap tive Le ngth R eduction ( AdapLeR ). Instead of relying on attention weights, our method trains a set of Contribution Predictors (CP) to estimate tokens' saliency scores at inference. We show that this choice results in more reliable scores than attention weights in measuring tokens' contributions. The most related study to ours is TR-BERT (Ye et al., 2021) which leverages reinforcement learning to develop an input-adaptive token selection policy network. However, as pointed out by the authors, the problem has a large search space, making it difficult for RL to solve. To mitigate this, they resorted to extra heuristics such as imitation learning (Hussein et al., 2017) for warming up the training of the policy network, action sampling for limiting the search space, and knowledge distillation for transferring knowledge from the intact backbone fine-tuned model. All of these steps significantly increase the training cost. Hence, they only perform token selection at two layers. In contrast, we propose a simple but effective method to gradually eliminate tokens in each layer throughout the training phase using a soft-removal function which allows the model to be adaptable to various inputs in a batch-wise mode. It is also worth noting in contrast to our approach above studies are based on top-k operations for identifying the k most important tokens during training or inference, which can be expensive without a specific hardware architecture (Wang et al., 2021). In summary, our contributions are threefold:  We couple a simple Contribution Predictor (CP) with each layer of the model to estimate tokens' contribution scores to eliminate redundant representations. Instead of an instant token removal, we gradually mask out less contributing token representations by employing a novel soft-removal function. We also show the superiority of our token selection strategy over the other widely used strategies by using human rationales. Self-attention is a core component of the Transformers (Vaswani et al., 2017) which looks for the relation between different positions of a single sequence of token representations ( x 1 , ..., x n ) to build contextualized representations. To this end, each input vector x i is multiplied by the corresponding trainable matrices Q , K , and V to respectively produce query ( q i ), key ( k i ), and value ( v i ) vectors. To construct the output representation z i , a series of weights is computed by the dot product of q i with every k j in all time steps. Before applying a softmax function, these values are divided by a scaling factor and then added to an attention mask vector m , which is zero for positions we wish to attend and  (in practice,  10000 ) for padded tokens (Vaswani et al., 2017). Mathematically, for a single attention head, the weight attention from token x i to token x j in the same input sequence can be written as:  i,j = softmax x j X (cid:32) q i k  j  d + m i (cid:33)  R (1) The time complexity for this is O ( n 2 ) given the dot product q i k  j , where n is the input sequence length. This impedes the usage of self-attention based models in low-resource settings. While self-attention is one of the most white-box components in transformer-based models, relying on raw attention weights as an explanation could be misleading given that they are not necessarily responsible for determining the contribution of each token in the final classifier's decision (Jain and Wallace, 2019; Serrano and Smith, 2019; Abnar and Zuidema, 2020). This is based on the fact that raw attentions are being faithful to the local mixture of information in each layer and are unable to obtain a global perspective of the information flow through the entire model (Pascual et al., 2021). Gradient-based methods provide alternatives to attention weights to compute the importance of a specific input feature. Despite having been widely utilized in other fields earlier (Ancona et al., 2018; Simonyan et al., 2013; Sundararajan et al., 2017; Smilkov et al., 2017), they have only recently become popular in NLP studies (Bastings and Fil-ippova, 2020; Li et al., 2016; Yuan et al., 2019). These methods are based on computing the first-order derivative of the output logit y c w.r.t. the input embedding h 0 i (initial hidden states), where c could be true class label to find the most important input features or the predicted class to interpret model's behavior. After taking the norm of output derivatives, we get sensitivity (Ancona et al., 2018), which indicates the changes in model's output with respect to the changes in specific input dimensions. Instead, by multiplying gradients with input features, we arrive at gradient  input (Bastings and Filippova, 2020), also known as saliency , which also considers the direction of input vectors to determine the most important tokens. Since these scores are computed for each dimension of embedding vectors, an aggregation method such as L2 norm or mean is needed to produce one score per input token (Atanasova et al., 2020a): S i =  y c h 0 i  h 0 i  2 (2) 3 Methodology As shown in Figure 1, our approach relies on dropping low contributing tokens in each layer and passing only the more important ones to the next. Therefore, one important step is to measure the importance of each token. To this end, we opted for saliency scores which have been recently shown as a reliable criterion in measuring token's contributions (Bastings and Filippova, 2020; Pascual et al., 2021). In Section 5.1 we will show results for a series quantitative analyses that supports this choice. In what follows, we first describe how we estimate saliency scores at inference time using a set of Contribution Predictors (CPs) and then elaborate on how we leverage these predictors during inference (Section 3.2) and training (Section 3.3). Computing gradients during inference is problematic as backpropagation computation prolongs inference time, which is contrary to our main goal. To circumvent this, we simply add a CP after each layer  in the model to estimate contribution score for each token representation, i.e.,  S i . The model then decides on the tokens that should be passed to the next layer based on the values of  S i . CP computes  S  i for each token using an MLP followed by a softmax activation function. We argue that, despite being limited in learning capacity, the MLP is sufficient for estimating scores that are more generalized and relevant than vanilla saliency values. We will present a quantitative analysis on this topic in Section 5. Most BERT-based models consist of L encoder layers. The input sequence of n tokens is usually passed through an embedding layer to build the initial hidden states of the model h 0 . Each encoder layer then produces the next hidden states using the 3 ones from the previous layer: h  = Encoder  ( h   1 ) (3) In our approach, we eliminate less contributing token representations before delivering hidden states to the next encoder. Tokens are selected based on the contribution scores  S  obtained from the CP of the corresponding layer  . As the sum of these scores is equal to one, a uniform level indicates that all tokens contribute equally to the prediction and should be retained. On the other hand, the lower-scoring tokens could be viewed as unnecessary tokens if the contribution scores are concentrated only on a subset of tokens. Given that the final classification head uses the last hidden state of the [CLS] token, we preserve this token's representation in all layers. Despite preserving this, other tokens might be removed from a layer when [CLS] has a significantly high estimated contribution score than others. Based on this intuition, we define a cutoff threshold based on the uniform level as:   =    1 / n with 0 <    1 to distinguish important tokens. Tokens are considered important if their contribution score exceeds  (which is a value equal or smaller than the uniform score). Intuitively, a larger  provides a higher  cutoff level, thereby dropping a larger number of tokens, hence, yielding more speedup. The value of  determines the extent to which we can rely on CP's estimations. In case the estimations of CP are deemed to be inaccurate, its impact can be reduced by lowering  . We train each layer's   using an auxiliary training objective, which allows the model to adjust the cutoff value to control the speedup-performance tradeoff. Also, since each input instance has a different computational path during token removal process, it is obvious that at inference time, the batch size should be equal to one (single instance usage), similarly to other dynamic approaches (Zhou et al., 2020; Liu et al., 2020; Ye et al., 2021; Eyzaguirre et al., 2021; Xin et al., 2020). Training consists of three phases: initial fine-tuning, saliency extraction, and adaptive length retraining. In the first phase, we simply fine-tune the backbone model (BERT) on a given target task. We then extract the saliencies of three top-perfroming checkpoints from the fine-tuning process and compute the average of them to mitigate potential inconsistencies in saliency scores (cf. Section 2.2). The final step is to train a pre-trained model using an adaptive length reduction procedure. In this phase, a non-linear function gradually fades out the representations throughout the training process. Each CP is jointly trained with the rest of the model using the saliencies extracted in the previous phase alongside with the target task labels. We also define a speedup tuning objective to determine the thresholds (via tuning  ) to control the performance-speedup trade-off. In the following, we elaborate on the procedure. Soft-removal function. During training, if tokens are immediately dropped similarly to the inference mode, the effect of dropping tokens cannot be captured using a gradient backpropagation procedure. Using batch-wise training in this scenario will also be problematic as the structure will vary with each example. Hence, inspired by the padding mechanism of self-attention models (Vaswani et al., 2017) we introduce a new procedure that gradually masks out less contributing token representations. In each layer, after predicting contribution scores, instead of instantly removing the token representations, we accumulate a negative mask to the attention mask vector M using a soft-removal function: m  i (  S i ) =    adj (  S i    )     S i <   (  S i  1)  (1    )   S i    (4) This function consists of two main zones (Figure 2). In the first term, the less important tokens with scores lower than the threshold (   ) are assigned higher negative masking as they get more distant 4 from  . The slope is determined by  adj =  /  , where  is a hyperparameter that is increased exponentially after each epoch (e.g.,   10   after finishing each epoch). Increasing  makes the soft-removal function stronger and more decisive in masking the representations. To avoid undergoing zero gradients during training, we define 0 <  < 0 . 1 to construct a small negative slope (similar to the well known Leaky-ReLU of Maas et al. 2013) for those tokens with higher contributing scores than   threshold. Consider a scenario in which   sharply drops, causing most of  S i get over the   threshold. In this case, the non-zero value in the second term of Equation 4, which facilitates optimizing   . Training the Contribution Predictors. The CPs are trained by an additional term which is based on the KL-divergence 1 of each layer's CP output with the extracted saliencies. The main training objective is a minimization of the following loss: L = LCE +  LCP (5) Where  is a hyperparameter which that specifies the amount of emphasis on the CP training loss: LCP = L  1 (cid:88)  =0 ( L   ) DKL (  S  ||  S  ) = L  1 (cid:88)  =0 ( L   ) N (cid:88) i =1  S i log(  S i  S i ) (6) Since S is based on the input embeddings, the [CLS] token usually shows a low amount of contribution due to not having any contextualism in the input. As we leverage the representation of the [CLS] token in the last layer for classification, this token acts as a pooler and gathers information about the context of the input. In other words, the token can potentially have more contribution as it passes through the model. To this end, we amplify the contribution score of [CLS] and renormalize the distribution (  S  ) with a trainable parameter   :  S i =   S  1 1 [ i = 1] + S i 1 [ i > 1]   S  1 + (cid:80) ni =2 S i (7) By this procedure, the next objective (discussed in the next paragraph) will have the capability of tuning the amount of pooling, consequently controlling the amount of speedup. Larger  push the 1 Inclusive KL loss. CPs to shift the contribution towards the [CLS] token to gather most of the task-specific information and avoids carrying redundant tokens through the model. Speedup Tuning. In the speedup tuning process, we combine the cross-entropy loss of the target classification task with a length loss which is the expected number of unmasked token representations in all layers. Considering that we have a non-positive and continuous attention mask M , the length loss of a single layer would be the summation over the exponential of the mask values exp( m i ) to map the masking range [  , 0] to a [0 (fully masked/removed) , 1 (fully retained) ] bound. LSPD ./P ERF . = LCE +  LLENGTHLLENGTH = L (cid:88) l =1 n (cid:88) i =1 exp( m  i ) (8) Equation 8 demonstrates how the length loss is computed inside the model and how it is added to the main classification loss. During training, we assign a separate optimization process which tunes  and  to adjust the thresholds and the amount of [CLS] pooling 2 alongside with the CP training. The reason that this objective is treated as a separate problem instead of merging it with the previous one, is because in the latter case the CPs could be influenced by the length loss and try to manipulate the contribution scores for some tokens regardless of their real influence. So in other words, the first objective is to solve the task and make it explainable with the CPs, and the secondary objective builds the speedup using tuning the threshold levels and the amount of pooling in each layer. To verify the effectiveness of AdapLeR on inference speedup, we selected eight various text classification datasets. In order to incorporate a variety of tasks, we utilized SST-2 (Socher et al., 2013) and IMDB (Maas et al., 2011) for sentiment, MRPC (Dolan and Brockett, 2005) for paraphrase, AG's News (Zhang et al., 2015) for topic classification, DBpedia (Lehmann et al., 2015) for knowledge extraction, MNLI (Williams et al., 2018) for NLI, 2 Since  is not in the computational DAG, we employed a dummy variable inside the model. QNLI (Rajpurkar et al., 2016) for question answering, and HateXplain (Mathew et al., 2021) for hate speech. 3 Evaluations are based on the test split of each dataset. For those datasets that are in the GLUE Benchmark (Wang et al., 2018), test results were acquired by submitting the test predictions to the evaluation server. As our baseline, we report results for the pre-trained BERT model (base-uncased) (Devlin et al., 2019) which is also the backbone of AdapLeR. We also compare against three other approaches: DistilBERT (uncased) (Sanh et al., 2019) as a static compression method, PoWER-BERT and TR-BERT as two strong length reduction methods (cf. Sec. 1). We used the provided implementations and suggested hyperparameters 4 to train these baselines. To fine-tune the backbone model, we used same hyperparameters over all tasks (see Section D for details). The backbone model and our model implementation is based on the Hug-gingFace's Transformers library (Wolf et al., 2020). Trainings and evaluations were conducted on a dual 2080Ti 11GB GPU machine with multiple runs. Hyperparameter Selection. Overall, we introduced four hyperparameters ( , , ,  ) 5 which are involved in the training process. Among these,  and  are the primary terms that have considerable effects on AdapLeR's downstream performance and speedup. This makes our approach comparable to existing techniques (Goyal et al., 2020; Ye et al., 2021) which usually have two or three hyperparameters adjusted per task. We used grid search to 3 See the statistics of datasets in Table 5 in Appendix. 4 Since some of the datasets were not used originally, we had to search the hyperparameters based on the given ranges. 5 Note that  and  are trainable terms that are tuned by the model during training. find the optimal values for these two terms, while keeping the other hyperparameters constant over all datasets. Hyperparamter selection is further discussed in Section D. FLOPs Computation. We followed Ye et al. (2021) and Liu et al. (2020) and measured computational complexity in terms of FLOPs, i.e., the number of floating-point operations (FLOPs) in a single inference procedure. This allows us to assess models' speedups independently of their operating environment (e.g., CPU/GPU). The total FLOPs of a given model is a summation of the measured FLOPs over all test examples. Then, a model's speedup can be defined as the total FLOPs measured on BERT (our baseline) divided by the corresponding model's total FLOPs. To have a fair comparison, we also computed FLOPs for PoWER-BERT in a single instance mode, described in Section C. 4.3 Results Table 1 shows performance and speedup for AdapLeR and other comparison models across eight different datasets. While preserving the same level of performance, AdapLeR outperforms other techniques in terms of speedup across all tasks (ranging from +0.2x to +7.4x compared to the best model in each dataset). It is noteworthy that the results also reveal some form of dependency on the type of the tasks. Some tasks may need less amount of contextualism during inference and could be classified by using only a fraction of input tokens. For instance, in AG's News, the topic of a sentence might be identifiable with a single token (e.g., soccer  Topic: Sports, see Figure 6 in the Appendix for an example). PoWER-BERT adopts attention weights in its token selection which requires at least one layer of computation to be determined, and TR-BERT ap-6 1.00 1.25 1.50 1.75 2.00 2.25 2.50 2.75 Speedup 0.86 0.87 0.88 0.89 0.90 0.91 0.92 0.93 A cc u r a c y SST-2 TR PoWER AdapLeR 2 3 4 5 6 7 Speedup 0.64 0.65 0.66 0.67 0.68 0.69 0.70 A cc u r a c y HateXplain TR PoWER AdapLeR Figure 3: Accuracy-Speedup trade-off curve for AdapLeR and two other state-of-the-art reduction methods; TR: TR-BERT, PoWER: PoWER-BERT on two different tasks. plies token elimination only in two layers to reduce the training search space. In contrast, our procedure performs token elimination for all layers of the model, enabling a more effective removal of redundant tokens. On the other hand, we observe that TR-BERT and PoWER-BERT lack any speedup gains for tasks such as QNLI, MNLI, and MRPC which need a higher degree of contextualism during inference. However, AdapLeR can offer some speedups even for these tasks. Speedup-Performance Tradeoff. To provide a closer look at the efficiency of AdapLeR in comparison with the other state-of-the-art length reduction methods, we illustrate speedup-accuracy curves in Figure 3. We provide these curves for two tasks in which other length reduction methods show comparable speedups to AdapLeR. For each curve, the points were obtained by tuning the most influential hyperparameters of the corresponding model. As we can see, AdapLeR significantly outperforms the other two approaches in all two tasks. An interesting observation here is that the curves for TR-BERT and AdapLeR are much higher than that of PoWER-BERT. This can be attributed to the input-adaptive procedure employed by the former two methods for determining the number of reduced tokens (whereas PoWER-BERT adopts a fixed retention configuration in token elimination). In this section, we first conduct an experiment to support our choice of saliency scores as a supervision in measuring the importance of token representations. Next, we evaluate the behavior of Contribution Predictors in identifying the most important tokens in the AdapLeR. A natural question that arises when dealing with token pruning is that of importance measure : what is the most appropriate criterion for assessing the relative importance of tokens within a sentence? We resort to human rationale as a reliable upper bound for measuring token importance. To this end, we used the ERASER benchmark (DeY-oung et al., 2020), which contains multiple tasks for which important spans of the input text have been highlighted as supporting evidence (aka ra-tionale) by human. Among the tasks in the benchmark, we opted for two diverse classification tasks: Movie reviews (Zaidan and Eisner, 2008) and MultiRC (Khashabi et al., 2018). In the former task, the model predicts the sentiment of the passage. Whereas the latter contains a passage, a question, and multiple candidate answers, which is cast as a binary classification task of passage/question/answer triplets in the ERASER benchmark. In order to verify the reliability of human rationales, we fine-tuned BERT based on the rationales only, i.e., by excluding those tokens that are not highlighted as being important in the input. In Table 2, the first two rows show the performance of BERT on the two tasks with full input and with human rationales only. We see that fine-tuning merely 7 SST-2 (dev) Label: Negative QNLI (dev) Label: Entailment Figure 4: The illustration of contribution scores obtained by CPs in three different layers of the model for two input examples from SST-2 (sentiment) and QNLI (Question-answering NLI) tasks. on rationales not only yields less computation cost, but also results in a better performance when compared with the full input setting. Obviously, human annotations are not available for a whole range of downstream NLP tasks; therefore, this criterion is infeasible in practice and can only be viewed as an upper bound for evaluating different strategies in measuring token importance. We investigated the effectiveness of saliency and self-attention weights as two commonly used strategies for measuring the importance of tokens in pre-trained language models. To compute these, we first fine-tuned BERT with all tokens in the input for a given target task. We then obtained saliency scores with respect to the tokens in the input embedding layer. This brings about two advantages. Firstly, representations in the embedding layer are non-contextualized, allowing us to measure the importance of each token independently from the others. Secondly, the backpropagation of gradients through layers to the beginning of the model provides us with aggregated values for the relative importance of each token based on the entire model. Similarly, we aggregated the self-attention weights across all layers of the model using a post-processed variant of attentions called attention rollout (Abnar and Zuidema, 2020), a popular technique in which the attention weight matrix in each layer is multiplied with the preceding ones to form aggregated attention values. To assign an importance score to each token, we examined two different interpretation of attention weights. The first strategy is the one adopted by PoWER-BERT (Goyal et al., 2020) in which for each token we accumulate attention values from other tokens. Additionally, we measured how much the [CLS] token attends to each token in the sentence, a strategy which has been widely used in interpretability studies around BERT (Abnar and Zuidema, 2020; Chrysostomou and Aletras, 2021; Jain et al., 2020, inter alia ). For a fair comparison, for each sentence in the test set, we selected the topk salient and attended words, with k being the number of words that are annotated as rationales. Results in Table 2 show that fine-tuning on the most salient tokens outperforms that based on the most attended tokens. This denotes that saliency is a better indicator for the importance of tokens. Nonetheless, recent length reduction techniques (Goyal et al., 2020; Kim and Cho, 2021; Wang et al., 2021) have mostly adopted attention weights as their criterion for selecting important tokens. Computing these weights is convenient as they are already computed during the forward pass, whereas computing saliency requires an additional backpropagation step. Note that in our approach, saliency scores are easily estimated within inference time by the pre-trained CPs. In this section we validate our Contribution Predictors in selecting the most contributed tokens. Figure 4 illustrates two examples from the SST-2 and QNLI datasets in which CPs identify and gradually drop the irrelevant tokens through layers, finally focusing mostly on the most important token representations; pedestrian (adjective) in SST-2 and tesla coil in the passage part of QNLI (both of which are highly aligned with human rationale). In order to quantify the extent to which AdapLeR's CPs can preserve rationales without requiring direct human annotations in an unsuper-8 0.1 0.2 0.3 FPR Saliency Attention Attention Rollout CP 1 2 3 4 5 6 7 8 9 10 11 12 Layer 0.35 0.40 0.45 0.50 m AP Figure 5: Agreement with human rationales in terms of mean Average Precision and False Positive Rate for Contribution Predictor (CP) and three alternative techniques. vised manner we carried out the following experiment. To investigate the effectiveness of trained CPs in predicting human rationales we computed the output scores of CPs in AdapLeR for each token representation in each layer. We also fine-tuned a BERT model on the Movie Review dataset and computed layer-wise raw attention, attention rollout, and saliency scores for each token representation. Since human rationales are annotated at the word level, we sum the scores across tokens corresponding to each word to arrive at word-level importance scores. In addition to these soft scores, we used the uniform-level threshold (i.e., 1 / n ) to reach a binary score indicating tokens selected in each layer. As for evaluation, we used the Average Precision (AP) and False Positive Rate (FPR) metrics by comparing the remaining tokens to the human rationale annotations. The first metric measures whether the model assigns higher continuous scores to those tokens that are annotated by humans as rationales. Whereas, the intuition behind the second metric is how many irrelevant tokens are selected by the model to be passed to subsequent layers. We used soft scores for computing AP and binary scores for computing FPR. Figure 5 shows the agreement between human rationales and the selected tokens based on the two metrics. In comparison with the other widely used strategies for selecting important tokens, such as salinecy and attention, our CPs have significantly less false positive rate in preserving rationales through layers. Despite having similar FPRs at the final layer, CP is preferable to attention in that it can better identify rationales at the earlier layers, allowing the model to combine the most relevant token representations when building the final one. This in turn results in better performance, as was also shown in the previous experiment in Section 5.2. Also, we see that the curve of mAP for saliency is consistently higher than other strategies in terms of alignment with human rationales which supports our choice of saliency as a measure for token importance. Finally, we note that there is a line of research that attempts at guiding models to perform humanlike reasoning by training rationale generation simultaneously with the target task that requires human annotation (Atanasova et al., 2020b; Zhao et al., 2020; Li et al., 2018). As a by-product of the contribution estimation process, our trained CPs are able to generate these rationales at inference without the need for human-generated annotations. In this paper, we introduced AdapLeR, a novel method that accelerates inference by dynamically identifying and dropping less contributing token representations through layers of BERT-based models. Specifically, AdapLeR accomplishes this by training a set of Contribution Predictors based on saliencies extracted from a fine-tuned model and applying a gradual masking technique to simulate input-adaptive token removal during training. Empirical results on eight diverse text classification tasks show considerable improvements over existing methods. Furthermore, we demonstrated that contribution predictors generate rationales that are highly in line with those manually specified by humans. As future work, we aim to apply our technique to more tasks and see whether it can be adapted to those tasks that require all token representations to be present in the final layer of the model (e.g., question answering). Additionally, combining our width-based strategy with a depth-based one (e.g., early exiting) might potentially yield greater efficiency, something we plan to pursue as future work. Using our proposed method, pre-trained language models can use fewer FLOPs, reducing energy use and carbon emissions (Schwartz et al., 2020a).
abstain abstain objective objective method result method objective other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective abstain abstain abstain abstain abstain objective abstain objective objective result other other other method other other other other other other other other method method other other abstain abstain objective abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain other abstain abstain abstain method method abstain abstain abstain abstain abstain abstain result other abstain abstain abstain method method method method method abstain abstain method abstain method method method abstain abstain abstain method method abstain method method abstain abstain abstain abstain other abstain method method abstain abstain method method abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain method method abstain abstain method abstain method abstain abstain abstain method abstain objective abstain abstain objective objective method abstain