Transformer architecture has become the defacto model for many machine learning tasks from natural language processing and computer vision. As such, improving its computational efficiency becomes paramount. One of the major computational inefficiency of Transformer-based models is that they spend the identical amount of computation throughout all layers. Prior works have proposed to augment the Transformer model with the capability of skimming tokens to improve its computational efficiency. However, they suffer from not having effectual and end-to-end optimization of the discrete skimming predictor. To address the above limitations, we propose the Transkimmer architecture, which learns to identify hidden state tokens that are not required by each layer. The skimmed tokens are then forwarded directly to the final output, thus reducing the computation of the successive layers. The key idea in Transkimmer is to add a parameterized predictor before each layer that learns to make the skimming decision. We also propose to adopt reparameterization trick and add skim loss for the end-to-end training of Transkimmer. Transkimmer achieves 10 . 97  average speedup on GLUE benchmark compared with vanilla BERT base baseline with less than 1% accuracy degradation. The Transformer model (Vaswani et al., 2017) has pushed the accuracy of various NLP applications to a new stage by introducing the multi-head attention (MHA) mechanism (Lin et al., 2017). Further, the BERT (Devlin et al., 2019) model advances its performances by introducing self-supervised pre-training, and has reached the state-of-the-art accuracy on many NLP tasks. Compared to the recurrent fashion models, e.g. RNN (Rumelhart et al., 1986), LSTM (Hochreiter and Schmidhuber, 1997), the Transformer model leverages the above attention mechanism to process Layer 1 Layer 2 Layer 3 Transkimmer TransformerLayers [CLS] It is a goodfilm . all the input sequence. By doing so, extremely large scale and long span models are enabled, resulting in a huge performance leap in sequence processing tasks. However, the computation complexity of the attention mechanism is O ( N 2 ) with the input length of N , which leads to the high computation demand of the Transformer model. Some prior works (Goyal et al., 2020; Kim and Cho, 2021; Kim et al., 2021; Ye et al., 2021) explore the opportunity on the dynamic reduction of input sequence length to improve the Transformer's computational efficiency. Its intuition is similar to the human-being's reading comprehension capability that does not read all words equally. Instead, some words are focused with more interest while others are skimmed. For Transformer models, this means adopting dynamic computation budget for different input tokens according to their contents. To excavate the efficiency from this insight, we propose to append a skim predictor module to the Transformer layer to conduct fine-grained dynamic token pruning as shown in Fig. 1. When processed by the Transformer layers, the sequence of token 7275 hidden state embeddings are pruned at each layer with reference to its current state. Less relevant tokens are skimmed without further computation and forwarded to the final output directly. Only the significant tokens are continued for successive layers for further processing. This improves the Transformer model inference latency by reducing the input tensors on the sequence length dimension. However, the optimization problem of such skim decision prediction is non-trivial. To conduct pruning of dynamic tensors, non-differentiable discrete skim decisions are applied. Prior works have proposed to use soft-masking approximation or reinforcement learning to resolve, which leads to approximation mismatch or nonuniform optimization. Transkimmer propose to adopt reparameterization technique (Jang et al., 2017) to estimate the gradient for skim prediction. As such, we can achieve the end-to-end joint optimization obejective and training paradigm. By jointly training the downstream task and skim objective, the Transformer learns to selectively skim input contents. In our evaluation, we show Transkimmer outperforms all prior input reduction works on inference speedup gain and model accuracy. Specifically, BERT base is accfelerated for 10 . 97  on GLUE benchmark and 2 . 81  without counting the padding tokens. Moreover, we also demonstrate the method proposed by Transkimmer is generally applicable to pre-trained language models and compression methods with RoBERTa, DistillBERT and ALBERT models. This paper contributes to the following 3 aspects. We propose the Transkimmer model which accelerates the Transformer inference with dynamic token skimming. We further propose an end-to-end joint optimization method that trains the skim strategy together with the downstream objective. We evaluate the proposed method on various datasets and backbone models to demonstrate its generality. Recurrent Models with Skimming. The idea to skip or skim irrelevant sections or tokens of input sequence has been studied in NLP models, especially recurrent neural networks (RNN) (Rumel-hart et al., 1986) and long short-term memory network (LSTM) (Hochreiter and Schmidhuber, 1997). When processed recurrently, skimming the computation of a token is simply jumping the current step and keep the hidden states unchanged. LSTM-Jump (Yu et al., 2017), Skim-RNN (Seo et al., 2018), Structural-Jump-LSTM (Hansen et al., 2019) and Skip-RNN (Campos et al., 2018) adopt this skimming design for acceleration in recurrent models. Transformer with Input Reduction. Unlike the sequential processing of the recurrent models, the Transformer model calculates all the input sequence tokens in parallel. As such, skimming can be regarded as the reduction of hidden states tensor on sequence length dimension. Universal Transformer (Dehghani et al., 2019) proposes a dynamic halting mechanism that determines the refinement steps for each token. DeFormer (Cao et al., 2020) proposes a dual-tower structure to process the question and context part separately at shallow layers specific for QA task. The context branch is preprocessed off-line and pruned at shallow layers. Also dedicated for QA tasks, Block-Skim (Guan et al., 2021) proposes to predict and skim the irrelevant context blocks by analyzing the attention weight patterns. Progressive Growth (Gu et al., 2021) randomly drops a portion of input tokens during training to achieve better pre-training efficiency. Another track of research is to perform such input token selection dynamically during inference, which is the closest to our idea. POWER-BERT (Goyal et al., 2020) extracts input sequence at token level while processing. During the fine-tuning process for downstream tasks, Goyal et al. proposes a soft-extraction layer to train the model jointly. Length-Adaptive Transformer (Kim and Cho, 2021) improves it by forwarding the inflected tokens to final downstream classifier as recovery. Learned Token Pruning (Kim et al., 2021) improves POWER-BERT by making its pre-defined sparsity ratio a parameterized threshold. TR-BERT (Ye et al., 2021) adopts reinforcement learning to independently optimize a policy network that drops tokens. Comparison to these works are discussed in detail in Sec. 3. Moreover, SpAttn (Wang et al., 2021) facilitate POWER-BERT design with a domain-specific hardware design for better acceleration and propose to make skimming decisions with attention values from all layers. Early Exit Early exit (Panda et al., 2016; Teer-apittayanon et al., 2016) is another method to execute the neural network with input-dependent computational complexity. The idea is to halt the execution during model processing at some early exits. Under the circumstance of processing sequential inputs, early exit can be viewed as a coarse-grained case of input skimming. With the hard constraint that all input tokens are skimmed at the same time, early exit methods lead to worse accuracy and performance results compared to input skimming methods. However, the early exit method is also generally applicable to other domains like convolutional neural networks (CNN). DeeBERT (Xin et al., 2020), PABEE (Zhou et al., 2020), FastBERT (Liu et al., 2020) are some recent works adopting early exit in Transformer models. Magic Pyramid (He et al., 2021) proposes to combine the early exit and the input skimming ideas together. Tokens are skimmed with fine-grained granularity following POWER-BERT design and the whole input sequence is halted at some early exits. Efficient Transformer. There are also many efforts for designing efficient Transformers (Zhou et al., 2020; Wu et al., 2020; Tay et al., 2020). For example, researchers have applied well studied compression methods to Transformers, such as pruning (Guo et al.), quantization (Wang and Zhang, 2020; Guo et al., 2022), distillation (Sanh et al., 2019), and weight sharing. Other efforts focus on dedicated efficient attention mechanism considering its quadratic complexity of sequence length (Kitaev et al., 2020; Beltagy et al., 2020; Za-heer et al., 2020) or efficient feed-forward neural network (FFN) design regarding its dominant complexity in Transformer model (Dong et al., 2021). Transkimmer is orthogonal to these techniques on the input dimension reduction. In this section, we discuss the challenges of dynamic input skimming idea in details. Moreover, we compare techniques and design decisions from prior works described in Tbl. 1. 3.1 Optimization Method The first challenge of input skimming is the optimization with discrete skimming decisions. In specific, the decision for pruning the hidden state tensors (i.e., reducing their sequence length) is Models Optimization Input Discard Strategy POWER-BERT Soft-Masking Attention Discard Searched (Goyal et al., 2020) LAT Soft-Masking Attention Forward Searched (Kim and Cho, 2021) LTP Soft-Masking Attention Discard Learned (Kim et al., 2021) TR-BERT RL Embedding Forward Searched (Ye et al., 2021) Transkimmer Reparameterize Embedding Forward Learned Table 1: Summary of prior token reduction works and their design choices including POWER-BERT, Length-Adaptive Transformer (LAT), Learned Token Pruning (LTP) and TR-BERT. a binary prediction. As such, the skim prediction model is non-differentiable and unable to be directly optimized by gradient back propagation. Prior works handle the discrete binary skimming decision by using a set of complicated training techniques, which we categorize in Tbl. 1. Soft-Masking. Some works (Goyal et al., 2020; Kim and Cho, 2021; Kim et al., 2021) propose to use the soft-masking training trick which uses a continuous value for predicting the skimming prediction. During the training process, the predicted value is multiplied to the hidden states embedding vectors so that no actual pruning happens. In the inference phase, this continuous skimming prediction value is binarized by a threshold-based step function. The threshold value is pre-defined or determined through a hyper-parameter search process. Obviously, there exists a training-inference paradigm mismatch where the actual skimming only happens at the inference time. Such a mismatch leads to a significant accuracy degradation. Reinforcement Learning. TR-BERT (Ye et al., 2021) proposes to use the reinforcement learning (RL) to solve the discrete skimming decision problem. It uses a separated policy network as the skimming predictor, and the backbone Transformer model is considered as the value network. At first, the backbone Transformer is fine-tuned separately. It then updates the skimming policy network by using the RL algorithm. This multi-step training paradigm is tedious. And training the backbone Transformer and skimming policy network separately is sub-optimal compared to the joint optimization paradigm. Moreover, the large search space of such RL objective is difficult to converge especially on small downstream datasets. Reparameterization. In this work, we propose to use the reparameterization technique to address the discrete skimming decision challenge. Its core idea is to sample the backward propagation gradient during training, whose details we describe in Sec. 4. The advantage of our method is that it enables the joint optimization of skim predictor and backbone Transformer model and therefore achieves the optimal solution. For example, we will later demonstrate in Fig. 4 that the different tasks or datasets prefer different layer-wise skimming strategies, which are learned by our method. We will further explain the results in Sec. 5.4. In our work, we also jointly consider other design choices regarding the skimming optimization, which includes the choice of input to the skimming module and how to deal with the skimmed input. We first explain the choices made by prior works, and then explain the choice of our method. Strategy. For the skimming optimization methods described above, there can be different strategies regarding the implementation details. Generally, the skimming strategy can be categorized into search-based or learning-based approach, as described in Tbl. 1. However, when applied to various downstream NLP tasks and datasets, the dynamic skimming scheme prefers different layer-wise strategies as we mentioned above. This layer-wise skimming characteristics makes the search-based approach not scalable and generally applicable. In contrast, our method enables the joint training of skimming strategy and downstream task , which leads to better skimming decisions with reference to both efficiency and accuracy. LTP is the only by prior works adopting learning-based method, which, however, uses the soft-masking approach and suffers from the training-inference mismatch. Input for Skimming. POWER-BERT, LAT and LTP treat the attention weight value as importance score and utilize it as the criterion for making the skimming decision. Compared to this value-based method (Guan et al., 2020), TR-BERT uses hidden state embeddings as input feature. In our work, we use the hidden state embeddings because they enclose contextual information of the corresponding input token. Our work shows that the joint training of skimming module and backbone Transformer model leads to that the embeddings also learn to carry features for skimming prediction. Skimming Tokens. For the tokens pruned dynamically by the skimming decision during processing, it is natural to remove them from all the successive layers. However, LAT and TR-BERT propose to forward such tokens to the final output of the Transformer encoder, which keeps the dimension of the Transformer output unchanged. Our work adopts the forward-based design because it is more friendly for the Transformer decoder module on downstream tasks. To predict which tokens to be pruned, we append an extra prediction module before each layer as shown in Fig. 2. This prediction module outputs a skimming mask M , which is used to gather the hidden state embedding H at the sequence length dimension. The pruned embedding is then feed to the Transformer layer as its input. In the skim mask, we use output 1 to denote remaining tokens and 0 to denote pruned tokens. The gathering operation is to select the input tensor with a provided mask. By optimizing this stand-alone skim module, syntactically redundant and semantically irrelevant tokens are skimmed and pruned. The proposed skim predictor module is a multilayer perceptron (MLP) network composed of 2 linear layers with a layer norm operation (Ba et al., 2016) and GeLU activation (Hendrycks and Gimpel, 2016). The activation function is an arbitrary function with discrete output as skim decision. This skim predictor introduces extra model parameters and computation overhead. However, both of them are very small compared to the vanilla Transformer model, which are about 7 . 9% and 6 . 5% respectively. We demonstrate later that the computation overhead of skim module is much smaller than the benefits brought by the reduction of input tensor through skimming. For the tokens pruned by the skim module at each layer, we forward the these pruned hidden state embeddings to the last Transformer layer. As such, the final output of the whole Transformer model is composed of token embeddings skimmed at all layers and the ones processed by all layers without being skimmed. And this output is used for classification layers on various downstream tasks. This makes the skimming operation also compatible for token classification tasks such as extractive question answering (QA) and named entity recognition (NER). This also restores the once abandoned information for downstream tasks. In the above discussion, we have described that Transkimmer can be easily augmented to a backbone model without modification to its current structure. Furthermore, Transkimmer is also capable to utilize the pre-trained model parameters and finetune the Transkimmer activated Transformer-based models on downstream tasks. With an extra skim loss appended to the optimization object, this fine-tuning process is also performed end-to-end without changing its origin paradigm. Skim Attention. In the training procedure, Transkimmer does not prune the hidden state tensors as it does in the inference time. Because the gathering and pruning operation of a portion of tokens prevents the back-propagation of their gradients. The absence of error signal from negative samples interference the convergence of the Transkimmer model. Therefore, we propose skim-attention to mask the reduced tokens in training instead of actually pruning them. The attention weights to the skimmed tokens are set to 0 and thus unreachable by the other tokens. By doing so, the remaining tokens will have the identical computational value as actually pruning. And the gradient signal is passed to the skim predictor module from the skim attention multiplication. Gumbel Softmax. Following the discussion in Sec. 3.1, the output decision mask of skim predictor is discrete and non-differentiable. To conquer this inability of back propagation, we use the reparameterization method (Jang et al., 2017) to sample the discrete skim prediction from the output probability distribution  i of the MLP. The gradient of the non-differentiable activation function is estimated from the Gumbel-Softmax distribution during back propagation. M ij = Activation (  ij ) ,for j = 0,1 = GumbelSoftmax (  ij ) = exp (( log (  ij ) + g ij ) / ) (cid:80) 1 k =0 exp (( log (  ik ) + g ik ) / ) (5) g ij are independent and identically sampled from Gumbel (0 , 1) distribution. is the temperature hyper-parameter controlling the one-hot prediction distribution. We take  = 0 . 1 for all experiments. To achieve better token sparsification ratio, we further add a skim loss term to the overall optimization objective as follows Loss skim = 1 L 1 (cid:88) L  1 sum ( M i ) len ( M i ) . (6) The skim loss is essentially the ratio of tokens remained in each layer thus representing the computation complexity speedup. By decreasing this objective, more tokens are forced to be pruned during processing. To collaborate with the original downstream task loss, we use a harmony coefficient  to balance the two loss terms. As such, the total loss used for training is formulated as Loss total = Loss downstream + Loss skim . (7) With the use of the previous settings, the Transkimmer model is trained end-to-end without any change to its original training paradigm. Unbalanced Initialization. Another obstacle is that skimming tokens during the training process makes it much unstable and decreases its accuracy performance. With the pre-trained language modeling parameters, the skim predictor module is random initialized and predicts random decisions. This induces significant processing mismatch in the backbone Transformer model, where all tokens are accessible. Consequently, the randomly initialized skim predictor makes the training unstable and diverged. We propose an unbalance initialization technique to solve this issue. The idea is to force positive prediction at first and learn to skim gradually. Generally, parameters are initialized by zero mean distribution as   N (0 ,  ) . We propose to initialize the bias vector of the last linear layer in the skim predictor MLP with unbalanced bias as where i stands for the bias vector for prediction 1 or 0. Consequently, the skim predictor tends to reserve tokens rather than skimming them when innocent. The mean value  0 of the unbalanced distribution set to 5 for all the experiments. Datasets. We evaluate the proposed Transkimmer method on various datasets. We use the GLUE(Wang et al., 2019) benchmark including 9 classification/regression datasets, extractive question answering dataset SQuAD-v2.0, and sequence classification datasets 20News (Lang, 1995), YELP (Zhang et al., 2015) and IMDB (Maas et al., 2011). These datasets are all publicly accessible and the summary is shown in Tbl. 2. The diversity of tasks and text contexts demonstrates the general applicability of the proposed method. Models. We follow the setting of the BERT model to use the structure of the Transformer encoder and a linear classification layer for all the datasets. We evaluate the base setting with 12 heads and 12 layers in prior work (Devlin et al., 2019). We implement Transkimmer upon BERT and RoBERTa pre-trained language model on downstream tasks. Baselines. We compare our work to prior token reduction works including POWER-BERT (Goyal et al., 2020), Length-Adaptive Transformer (LA-Transformer) (Kim and Cho, 2021), Learned Token Pruning (LTP) (Kim et al., 2021), DeFormer (Cao et al., 2020) and TR-BERT (Kim et al., 2021). We also compare our method with model compression methods of knowledge distillation and weight sharing. Knowledge distillation uses a teacher model to transfer the knowledge to a smaller student model. Here we adopt DistilBERT (Sanh et al., 2019) setting to distill a 6-layer model from the BERT base model. By sharing weight parameters among layers, the amount of weight parameters reduces. Note that weight sharing does not impact the computa-7280 Method Padding COLA RTE QQP MRPC SST-2 MNLI WNLI QNLI STS-B Matthews FLOPs Acc. tion FLOPs (floating-point operations). We evaluate Transkimmer on ALBERT (Lan et al., 2020) that shares weight parameters among all layers. To express that token reduction method is compatible with these model compression methods, we further implement Transkimmer method with this works to demonstrate their cooperation effect. Besides, Dee-BERT(Xin et al., 2020) is a Transformer early exit baseline which can be regarded as coarse-grained input skimming. Padding. While processing batched input samples, Transformer models perform a padding operation on the input sequences to align the input length. Sequences are appended with a special padding token [PAD] to a predefined sequence length for the convenience of successive computing. This is a trivial setting for general evaluation but could lead to possible pseudo speedup for token reductions works. Because the padded tokens can be pruned without prediction. For the prior works, there are three evaluation settings with reference to padding, padding to a fixed sequence length, padding to mini-batch maximum length and no padding (denoted as Sequence , Batch and No in Fig. 3 & 4). We indicate the padding methods of prior works and evaluate Transkimmer with different padding settings for a fair comparison. The speedup of padding to mini-batch maximum length setting is related to batch size and processing order of input samples. So it is difficult to make a direct comparison under this setting. However, it can be estimated with padding to fixed sequence length as upper bound and no padding as lower bound. The sequence length on different datasets is determined following prior works' settings (Goyal et al., 2020; Kim et al., 2021). We measure the inference FLOPs as a general measurement of the model computational complexity on all platforms. We use the TorchProfile( ? ) tool to calculate the FLOPs for each model. Training Setting. We implement the proposed method based on open-sourced library from Wolf et al. (2020) 1 . For each baseline model, we use the released pre-trained checkpoints 2 . We follow the training setting used by Devlin et al. (2019) and Liu et al. (2019) to perform the fine-tuning on the above datasets. We perform all the experiments reported with random seed 42. We use four V100 GPUs for training experiments. The harmony coefficient  is determined by hyper-parameter grid search on development set with 20% data random picked from training set set. The search space is from 0 . 1 to 1 with a step of 0 . 1 . We show the overall results on several datasets and demonstrate our observations. Tbl. 3 demonstrates the accuracy and speedup evaluated on GLUE benchmark. And Tbl. 4 further demonstrates the results on other datasets with longer input. Comparison to vanilla model baseline. Generally, Transkimmer achieves considerably speedup to the vanilla models with a minor accuracy degradation, which is less than 1% for nearly all cases. The average speedup is 2 . 81  on GLUE benchmark and over 2  on the other datasets. This demonstrates the inference efficiency improvement of the Transkimmer input reduction method. We also evaluate Transkimmer with RoBERTa model as backbone and reach 3 . 24  average speedup on GLUE benchmark. This result further expresses the general applicability of Transkimmer with different Transformer-based pre-trained language models. Among all the datasets we evaluated, Transkimmer tends to have better acceleration ratio on the easier ones. For example, sequence classification tasks like QQP and STS-B are better accelerated than QA or NLI datasets. We suggest that the Transformer backbone is able to process the information at shallower layers and skim the redundant part earlier. This is also demonstrated in the following post-hoc analysis Sec. 5.4. Comparison to input reduction prior works. As shown in Tbl. 3, Transkimmer outperforms all the input reduction methods by a margin on GLUE benchmark. To make a fair comparison, we evaluate Transkimmer with two padding settings, padding to fixed sequence length or no padding. For most cases, Transkimmer has better accuracy performance and higher speedup ratio at the same time. When taking the special padding token into account, Transkimmer is able to accelerate 1 2 3 4 5 6 7 8 9 10 11 12 Layer 0 20 40 60 80 100 N u m b e r o f T o k e n s ( % ) MRPC WNLI STS-B COLA SST-2 QNLI QQP RTE MNLI Figure 4: Layer-wise skim strategies analysis of datasets from GLUE benchmark. BERT base model for 10 . 97  on GLUE benchmark. Transkimmer also outperforms the other methods on tasks shown in Tbl. 4. TR-BERT has the closet performance compared with Transkimmer but with a much complicated RL paradigm and larger search space. Comparison to model compression methods. The comparison to two model compression methods is shown in Tbl. 3. Transkimmer outperforms the knowledge distillation and weight sharing baseline by a margin. Besides, the dynamic skimming idea itself is orthogonal to this existing model compression methods. To elaborate, we further adopt the proposed Transkimmer method on DistilBERT and ALBERT models. With the proposed end-to-end training objective, Transkimmer is easily augmented to these methods. There is also no need to change the original training process. The result shows that the Transkimmer method further accelerates the inference efficiency of compressed models with nearly no extra accuracy degradation. Fig. 3 demonstrates the accuracy and performance trade-off analysis by tuning the harmony coefficient. We show the results on MRPC and SQuAD-v2.0 datasets to give comparisons with different baselines. It is shown that Transkimmer achieves a better accuracy to speedup Pareto curve compared to prior works. Transkimmer is able to provide better acceleration gain with less accuracy degradation. Especially, Transkimmer has a 1 . 5  speedup without accuracy loss. The result validates our design decisions analyzed in the input reduction search space choices. Skim Strategy. Fig. 4 is the result of the number of tokens remained for the processing of each Transformer layer. The normalized area under each curve is a rough approximation of the speedup ratio with reference to the tokens number. By end-to-end optimization, Transkimmer learns significant distinguished strategies on different tasks. On WNLI dataset, over 90% of tokens are pruned within the first 3 layers and guarantees a high acceleration gain. The steer cliff at layer 7 on COLA demonstrates a large portion of skimming at this particular position. We suggest that this is because the processing of contextual information is sufficient for the skimming decision at this specific layer. Post-Hoc Case Study. Moreover, several posthoc case studies are demonstrated with Tbl. 5. In the SST-2 sentimental analysis example, the defi-nite articles and apostrophes are discarded at the beginning. And all words are encoded in contextual hidden states embeddings and gradually discarded except for a few significant key words. Only the special token [CLS] is fully processed in this example for final sentimental classification. However, on the token classification task example from SQuAD dataset, all tokens are given to the downstream classifier to predict the answer position. The answer tokens are processed by all Transformer layers. Similarly, the question part is also kept with tokens containing enough information. Another detail worth mentioning is that we use subword tokenization for the SQuAD dataset. As such, subword tokens of the same word might be discarded at different layers. For instance, the word Francia is tokenized into franand -cia two subword tokens, which are pruned at layer 4 and 6 respectively. Input skimming or dynamic input reduction is an emerging Transformer model acceleration method studied by many works recently. This idea utilizes the semantic structure of language and the syntactic information of the input context for inference acceleration. Compared to static model weight compression methods, input skimming explores the redundancy in the input and hidden state tensors. As such, it is orthogonal and compatible with those model compression algorithms with its dynamic feature. In this work, we propose an accurate and efficient Transformer inference acceleration method by teaching it how to skim input contents. The proposed Transkimmer method is trained with an easy and end-to-end paradigm. Furthermore, Transkimmer is also generally applicable to various Transformer-based model structures. It is even compatible with the static model compression methods like knowledge distillation and weight sharing. We believe that the above features guarantee the Transkimmer method a wide range of applicable production scenarios. This work was supported by the National Key R&D Program of China under Grant 2021ZD0110104, the National Natural Science Foundation of China (NSFC) grant (U21B2017, 62106143, 62072297, and 61832006), and Shanghai Pujiang Program. We would like to thank the reviewers of ACL rolling review for their supportive comments and suggestions. Jingwen Leng and Minyi Guo are the corresponding authors of this paper.
abstain abstain abstain abstain abstain objective abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain objective objective objective objective objective other other other other other other other other other other other other method other other other other other other other other other other other other other other other other other other other other abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method method method abstain method method abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain method method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain objective abstain abstain abstain method other other other