Transformer-based models generally allocate the same amount of computation for each token in a given sequence. We develop a simple but effective token dropping method to accelerate the pretraining of transformer models, such as BERT, without degrading its performance on downstream tasks. In particular, we drop unimportant tokens starting from an intermediate layer in the model to make the model focus on important tokens more effi-ciently if with limited computational resource. The dropped tokens are later picked up by the last layer of the model so that the model still produces full-length sequences. We leverage the already built-in masked language modeling (MLM) loss to identify unimportant tokens with practically no computational overhead. In our experiments, this simple approach reduces the pretraining cost of BERT by 25% while achieving similar overall fine-tuning performance on standard downstream tasks. Nowadays, the success of neural networks in a variety of NLP tasks heavily relies on BERT-type language models containing millions to billions of parameters. However, the pretraining process of these models is computationally expensive, generating significant carbon dioxide emission (Strubell et al., 2019; Patterson et al., 2021). In practice, there is the need to perform large-scale language model pretraining for diverse applications (Lee et al., 2020; Chalkidis et al., 2020; Zou et al., 2021; Rogers et al., 2020) in different languages (Antoun et al., 2020; Sun et al., 2021). In this paper, we develop a technique that significantly reduces the pretraining cost of BERT models (Devlin et al., 2019) without hurting their test performance on a diverse set of fine-tuning tasks. Recent efforts of efficient training involve mixed-precision training (Shoeybi et al., 2019), distributed training (You et al., 2020), better modeling on rare words and phrases (Wu et al., 2021), designing more effective and data-efficient pretraining objectives (Lan et al., 2020; Clark et al., 2020; Raffel et al., 2020), progressive stacking (Gong et al., 2019), and so on. While these approaches contribute to efficient training with reduced computational cost, most of them focus on the model architecture or the optimization process. In this paper, we focus on a simple but efficient BERT-pretraining strategy that has been under-explored before, i.e., token dropping, which removes the redundant tokens in each sequence that are less informative to training. Since not all tokens contribute equally to the output or the training objective, and the computational complexity of transformer-based models grows at least linearly with respect to the sequence length, shortening the input sequences can accelerate the training effectively. Among existing studies, the depth-adaptive transformer approach aims to reduce the autoregressive inference time by allocating less computation on easy-to-predict tokens (Elbayad et al., 2020). To improve the training efficiency, Dai et al. (2020) perform pooling on the embeddings of nearby tokens. However, directly dropping tokens during pretraining was not studied until very recently in faster depth adaptive transformer (Liu et al., 2021), where the important tokens are identified either through (1) mutual information-based estimation between tokens and predefined labels or through (2) a separate BERT model that exhaustively computes the masked language model (MLM) loss for each token. On the contrary, we focus on accelerating the task-agnostic pretraining phase without requiring any labels or any computation by a separate language model. Specifically, we identify important tokens as the ones hard to predict by the model 3774 itself through its loss during training, which is adaptive to its training process and leads to practically no computational overhead. We show examples of dropped tokens in Figure 1. Recent approaches such as RoBERTa (Liu et al., 2019) suggest packing input sequences. In this way, there are no [PAD] tokens, which makes it a non-trivial task to identify unimportant tokens. We identify unimportant tokens in each sequence with the smallest historical MLM loss (we take the running average of the MLM loss of each token throughout the pretraining process). By removing them from intermediate layers of a BERT model during training, we save an enormous amount of computation and memory. We keep them in the first several layers as well as in the last layer so that they are still present in the model. Therefore, the inputs and outputs of BERT model are kept consistent with the conventional all-token training process. Without modifying the original BERT architecture or training setting, this simple token-dropping strategy trains intermediate layers mainly on a few important tokens. As demonstrated in our experiments, models pretrained in this way generalize well on diverse downstream tasks with full sequences. To summarize, our contributions are as follows. (1) We show that BERT models can be pretrained with only a subset of the layers focusing on important tokens. Even though the model is trained on sub-sequences of important tokens only, it generalizes well to full sequences during fine-tuning on downstream tasks. (2) We identify important tokens through the pretraining process by exploring the training dynamics, with minimal computational overhead and without modifying the model architecture. (3) We show that our token dropping strategy can save 25% of pretraining time while achieving similar performance on downstream tasks. Code is available at https://github.com/ tensorflow/models/tree/master/ official/projects/token_dropping . Recall that a sequence in BERT consists of two sentences as well as the classification token [CLS] and the separator token [SEP] . If the resulting number of tokens is smaller than 512, then padding tokens are added to ensure that each sequence is Does(1.1) the(0.3) experiment (3.2) look (1.3) good (2.10) ?(0.6) They(1.0) smell (2.1) something(1.8) horribly (5.4) wrong (2.3) .(0.2) Saturn (2.9) is(0.4) the(0.3) sixth (3.0) planet (2.4) from (1.0) the(0.3) sun (1.9) and(0.6) the(0.3) second (1.8) -(0.8) largest (1.4) in(0.5) the(0.3) solar (2.3) system (2.2) ,(0.4) after (1.5) jupiter (3.3) .(0.2) It(0.7) is(0.4) a(0.5) gas (2.2) giant (4.0) with (0.9) an(0.8) average (1.2) radius (3.3) of(0.3) about (1.3) nine (3.5) and(0.6) a(0.5) half (2.0) times (1.6) that(0.8) of(0.3) earth (2.4) .(0.2) We decide to use sequence packing (Liu et al., 2019) so that there would be no [PAD] symbols, throughout the paper. We also remove the next-sentence prediction training criteria as well. The rationale for using sequence-packing is two-fold. First, sequence packing provides a competitive baseline in terms of pretraining efficiency (So et al., 2019; Liu et al., 2019; Kosec et al., 2021; Zhang et al., 2021). Second, using sequence-packing can stress-test our algorithm under the absence of padding symbols to see if it brings further improvements beyond dropping padding tokens: without sequence packing, our algorithm can label [PAD] as the unimportant tokens, which trivially improves pretraining efficiency; with sequence packing, however, our algorithm has to identify and drop real tokens as unimportant tokens to improve the efficiency. Define T to be the input sequence length and d k , d v to be the size of each individual key and value vector respectively. The multi-head attention function with h attention heads is defined as: MultiHeadAttention( Q, K, V ) = concat( H 1 , . . . , H h ) WO , where H i = Attention( QW Qi , KW Ki , V W Vi ) = softmax (cid:32) ( QW Qi )( KW Ki ) (cid:62)  d k (cid:33) V W Vi . Use d model to denote the hidden size of the model (usually equal to hd k ). We have the following: Q, K, V  RT  d model , W Qi  R d model  d k , W Ki  R d model  d k , W Vi  R d model  d v , W Oi  R hd v  d model . Besides the attention sub-layer, each BERT encoder layer also contains the feed-forward sub-layer (or feed-forward network, abbreviated as FFN). Each FFN is a position-wise function: it is applied to each position of the input, identically. The input to FFN is a matrix  RT  d model . The input will be transformed by a two-layer perceptron (with ReLU activation in between) into an output matrix  RT  d o . In Vaswani et al. (2017), d o = d model and the hidden size of the intermediate layer d ff = 4  d model due to empirical investigation. Suppose the input sequence contains 512 tokens. Having 512 hidden states (corresponding to 512 tokens) after each encoder layer may not be necessary, given that certain words may never heavily influence the meaning of the sentence. 3 Moreover, removing unimportant tokens in intermediate layers would produce a dropout effect, so that our network would learn to reconstruct the masked words with more noisy hidden states. Therefore, we decide to allocate the full amount of computation only to important tokens. Figure 2 gives an illustration of where the unimportant tokens are dropped in a BERT model. Each row of query Q , key K , and value V in a self-attention module in each transformer encoder layer corresponds to a single token. Suppose L f = L full is the set of layers whose input covers all the tokens; 4 L h = L half is the set of layers whose input only cover a proper subset of tokens. 3 Relatedly, Zhang et al. (2019) have shown in computer vision, using fully-connected networks and convolutional neural networks, that certain layers called ambient layers can be reset with almost no negative consequence on performance, while other layers called critical layers are necessary. 4 In this paper, input covers all the tokens means that the query, key, and value metrics in the layer have T rows, so no rows are discarded from the matrices. Separation. During stage-1 pretraining, if the layer l  L f and the next layer l +1  L h , then we remove the rows in Q corresponding to the unimportant tokens for layer l + 1 but keep K and V intact. After the removal, we have Q  RM  d model where M is the number of important tokens. We also have K, V  RT  d model where T is the input sequence length. 5 Suppose l (cid:48) is the first layer above layer l +1 such that l (cid:48)  L f . Suppose l + 2  L h . Then, for layers l + 2 , . . . , l (cid:48)  1 , we have Q, K, V  RM  d model , which means that their rows correspond to only the important tokens. Merging. Given that l (cid:48) is the first layer above layer l + 1 such that l (cid:48)  L f , before layer l (cid:48) , we 5 In practice, using TensorFlow, the separation step in stage-1 pretraining and the merge step can be done using the function tf.gather() . The number of important tokens for different sequences has to be the same in order to use modern accelerators like TPUs. Using sparse tensors can address the issue of having a different number of important tokens, but sparse tensor operations in practice are slow. merge the hidden states corresponding to the unimportant tokens (taken from the outputs of layer l ) with the hidden states corresponding to the important tokens (taken from the outputs of layer l (cid:48)  1 ). We keep the order of hidden states consistent with the order of the input tokens. Alternatively: token passing instead of token dropping. In layers where unimportant tokens are dropped, the input to the layers effectively corresponds to partial and incoherent sentences. We thus attempt the token passing approach, which can ensure that the input to such layers corresponds to complete and coherent sentences. Token passing is described as follows. In layers l + 1 , . . . , l (cid:48)  1  L h , we can keep the rows of K and V corresponding to the unimportant tokens. More specifically, the rows of K that correspond to important tokens come from the hidden states outputted by the previous encoder layer. The rows of K that correspond to unimportant tokens come from the hidden states outputted by layer l . This procedure results in Q  RM  d model and K, V  RT  d model for layers l + 1 , . . . , l (cid:48)  1 . See Section 5 for empirical studies. Determining l and l (cid:48) . We leave details on determining l and l (cid:48) to later sections. Empirically, l = LE 2  1 and l (cid:48) = LE  1 consistently lead to good performance, where LE is the total number of encoder layers. For instance, if LE = 12 , then the full layers in L f (i.e., layers in which the query, key, and value matrices all have T rows) would be layers 1 through 5 as well as layer 12. At test-time or when we fine-tune on downstream tasks, all the encoder layers are full layers, meaning meaning we do not do any token dropping. Given the mismatch between the neural network in stage 1 and the neural network used for fine-tuning and test-time, during stage 2, we simply pretrain using the full model (i.e., all tokens passing through all layers). Stage-2 pretraining requires only a smaller number of steps, compared to stage-1 pretraining. However, stage-2 pretraining turns out to be unnecessary, which we discuss in later sections. In this subsection, we elaborate on which tokens to drop (i.e., which corresponding rows to discard in the query, key, and value matrices) in a given sequence. First, we never drop special tokens including [MASK] , [CLS] , and [SEP] . In other words, we always treat these tokens as important tokens. Recall that we use sequence packing in all of our experiments, unless noted otherwise. Therefore, there are no padding tokens [PAD] . 6 We introduce two approaches for identifying important tokens in the following sub-sections. In the ablation studies (Section 4.2), we will introduce more straightforward approaches as baselines. Updating the cumulative loss vector. We use a vector m  R |V| to approximate the difficulty of learning a specific token in the vocabulary V . The vector m is updated throughout the pretraining. Recall that BERT pretraining involves the masked language modeling (MLM) objective, where the model is asked to predict the tokens of the masked-out input tokens. Suppose n tokens in a sequence are masked out, then we would obtain n MLM negative log-likelihood (NLL) losses. For each token, we update the corresponding entry in the cumulative loss vector as follows: m i    m i + (1   )  (cid:96) i , (1) where (cid:96) i is the NLL loss that corresponds to the token i and   (0 , 1) is a coefficient that is close to 1. In particular, we never update the cumulative losses corresponding to the aforementioned special tokens ( [MASK] , [CLS] , and [SEP] ). The losses for those tokens are set to a large number such as 10 4 . 7 6 If we do not use sequence packing, we would always drop the [PAD] tokens. Deciding which tokens are unimportant. We need to drop the rows in the query, key, and value matrices corresponding to the unimportant tokens. To decide which tokens will be treated as unimportant ones, given a sequence of 512 tokens, we simply look up the 512 corresponding cumulative losses using m , and label the tokens that correspond to the smallest cumulative losses as unimportant tokens. In other words, suppose we have a sequence x = ( x 1 , x 2 , . . . , x T ) where T is the sequence length. Use [ T ] to denote { 1 , 2 , . . . , T } . Suppose  : [ T ]  [ T ] is a function such that x  (1) , x  (2) , . . . , x  ( T ) are the tokens sorted in decreasing order of the aforementioned cumulative loss. Then, we are treating x  (1) , . . . , x  ( M ) as important tokens (i.e., the tokens to keep), where M is a positive integer (e.g., M = int( T/ 2) ). We are treating x  ( M +1) , . . . , x  ( T ) as unimportant tokens. Optionally: adding randomness. We can optionally assign every token with a nonzero probability to be selected as an important token, which can potentially make the model generalize well on full sequences. For example, let J = int(0 . 05 T ) , given x  (1) , x  (2) , . . . , x  ( T ) as described above, we replace the last J important tokens x  ( M  J +1) , . . . , x  ( M ) with J tokens randomly chosen from x  ( M  J +1) , . . . , x  ( T ) . Then, the J randomly chosen tokens will be treated as important tokens. In later sections, we will empirically investigate whether the randomness is helpful. Before the start of pretraining, we count the number of occurrences of each token in the vocabulary V . During pretraining, given a sequence, suppose there are s special tokens. This approach assigns the special tokens as well as the M  s tokens that correspond to the lowest frequency as important tokens, where M is the target number of important tokens in a sequence. It treats the rest of the tokens as unimportant tokens. the padding token has the smallest lossgiven that NLL loss is always non-negative for all other tokens. We use the sequence-packed version of the dataset (Section 2.1) so as to ensure that we have to drop meaningful tokens instead of the [PAD] tokens. Downstream tasks. We fine-tune on GLUE tasks (Wang et al., 2018), whose datasets are on the larger end. We only use the 6 largest GLUE datasets: MNLI, where we use MNLI-m to denote MNLI-matched and MNLI-mm to denote MNLI-mismatched (Williams et al., 2018), QNLI (Ra-jpurkar et al., 2016), QQP 8 , SST (Socher et al., 2013), and the GLUE diagnostics set AX (Wang et al., 2018). Additionally, we also experiment on the question answering datasets: SQuAD v1.1 (Ra-jpurkar et al., 2016) and SQuAD v2.0 (Rajpurkar et al., 2018). The evaluation metric for each task can be found in Table 1. By default, the total training steps of each model is 1 million, using the settings in Section 4.4. We experiment with the following models. First, we have the baseline models. baseline (no sequence packing) : The original BERT with the non-sequence-packed input. baseline : The original BERT with the sequence-packed input. baseline (75% steps) : The original BERT with the sequence-packed input but only trained for 75 % of the steps. This baseline is trained using a similar amount of computation as our proposed token dropping methods. Next, we have the following methods that aim to save pretraining time. For token dropping methods, we drop 50% of the tokens (unless mentioned otherwise) in order to compare with the average pooling method (Dai et al., 2020) which reduces the sequence length by half. token drop : We perform stage-1 pretraining using the cumulative-loss token-dropping for 1M steps. token drop (rand) : Similar to the token drop method, except that we randomly drop 50% non-special tokens in a sequence, instead of dropping unimportant tokens. Special tokens like [CLS] and [SEP] are not dropped. 8 https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs 3778  token drop (half-rand) : It is similar to the token drop method except adding extra randomness to the important token selection, as introduced in Section 3.3. This half-random method can be viewed as a combination of token drop and token drop (rand).  token drop (layer rearranged) : It is similar to the token drop method except moving the last layer that processes all tokens to the beginning of the model. In other words, the layers in Figure 2 are rearranged such that full-sequence layers are only at the bottom. token drop (freq) : Similar to the token drop method, except that we identify important tokens using the frequency-based token-dropping scheme, as discussed in Section 3.3.2. token avg : Similar to the token drop method, except that we use average-pooling to compress the sequences instead of token drop. Suppose layer l  L f and the immediate next layer l (cid:48)  L h , as described in Section 3. Instead of dropping rows of the query, key, and value matrices, we apply average pooling with a window size of 2 and a stride of 2. In other words, suppose q 1 , . . . , q T are the rows of the query vector. Then, the T/ 2 new query vectors are ( q 1 + q 2 ) / 2 , ( q 3 + q 4 ) / 2 , . . . , ( q T  1 , q T ) / 2 , assuming that T is an even number. This idea is introduced in Funnel transformer (Dai et al., 2020). We also experiment with adding the optional stage-2 pretraining phase to the methods described above. In such cases, we first perform stage-1 pretraining for 900k steps and then stage-2 pretraining for 100k steps. To distinguish between the stage-1 only methods, we add + stage-2 at the end of the method description. The BERT architectures are the same as the ones in Devlin et al. (2019). We experiment on both BERT-base and BERT-large. For each BERT architecture, we train with two different input sequence lengths: 512 and 128. We use the sequence-packed input data, unless otherwise noted. We use TPUv3 to pretrain the BERT models. The batch size of each pretraining step is 512. We train each BERT model for 1 million steps. We use the AdamW optimizer (Loshchilov and Hutter, 2019). We adopt a peak learning rate of 1 e  4 and use the linear decay scheduler for the learning rate. We conduct extensive hyperparameter tuning for downstream tasks. For all GLUE tasks, we test different numbers of training epochs   { 2 , 3 , 4 , 5 , 6 , 8 , 10 } and peak learning rate values   { 5 e  6 , 1 e  5 , 2 e  5 , 3 e  5 , 4 e  5 } using the baseline pretrained BERT model. { 3 , 6 } and   { 1 e  5 , 2 e  5 } give the best overall results. Thus, for every pretrained model, we finetune on each individual GLUE task using the combinations of the two best  and  values (four settings in total) and take the best validation result. For SQuAD tasks, we test   { 1 , 2 , 3 , 4 , 5 , 6 , 8 } and   { 5 e  5 , 6 e  5 , 8 e  5 , 1 e  4 , 1 . 2 e  4 } using the baseline pretrained BERT model and find out that   { 4 , 8 } and   { 2 e  5 , 4 e  5 } produce the best results overall. Thus, we fine-tune every model with these settings and report the best validation result. We apply the linear decay learning rate schedule that ends with zero for all experiments. For each method, we pretrain two models with different random seeds. Then these two models are fine-tuned separately on individual downstream tasks. We then report the averaged result as the final result for each task. Table 1 shows the ablation study. As mentioned, each number in the table corresponds to the average performance of two pretrained models (using different random seeds) that are then separately fine-tuned. On whether stage-2 pretraining is useful. There is a mismatch between the neural network in stage-1 pretraining and the neural network used for fine-tuning and test-time. Therefore, we propose stage-2 pretraining where there is no token dropping so as to address the train-test mismatch. Comparing token drop with token drop + stage-2 in Table 1, we see that the performance of the model trained without stage-2 pretraining and the model trained with stage-2 pretraining perform sim-3779 Methods BERT-base AX MNLI-mm MNLI-m QNLI QQP SST GLUE-avg SQuAD SQuAD SQuAD (corr.) (acc.) (acc.) (acc.) (F1) (acc.) -v1 (F1) -v2 (F1) -avg baseline (no sequence packing) 76.36 84.61 84.28 91.56 90.94 95.73 87.25 90.11 78.89 84.50 baseline 76.52 84.47 84.44 90.58 90.97 96.18 87.19 89.71 79.00 84.35 baseline (75%) 76.38 84.43 84.36 90.21 90.82 96.00 87.04 89.33 78.14 83.73 proposed token drop 77.77 85.28 85.20 91.25 91.00 95.54 87.67 90.44 81.09 85.77 token drop + stage-2 77.70 84.91 85.04 91.40 91.00 95.98 87.67 90.32 79.90 85.11 token drop (half-rand) 77.08 84.92 84.81 91.36 90.94 96.80 87.65 90.34 80.38 85.36 token drop (half-rand) + stage-2 77.25 85.19 84.89 91.52 90.67 94.94 87.41 90.47 79.81 85.14 token drop (rand) + stage-2 76.88 84.56 84.56 91.27 90.78 95.65 87.28 89.65 78.61 84.13 token drop (freq) + stage-2 76.19 84.35 84.27 91.05 90.80 96.48 87.19 89.38 77.32 83.35 token avg + stage-2 76.92 84.83 84.69 90.94 90.89 97.03 87.55 90.23 79.35 84.79 token pass + stage-2 77.04 84.58 84.86 91.36 90.89 95.67 87.40 89.98 79.85 84.92 token drop (layer rearranged) + stage-2 76.61 84.52 84.37 90.78 90.76 96.65 87.28 90.05 78.38 84.21 Table 1: Evaluating different pretraining methods by finetuning pretrained models on downstream tasks. ilarly. We hypothesize that the train-test mismatch can be easily addressed during downstream task fine-tuning. On determining which tokens are important. Figure 1 shows which tokens are labeled as important using three examples from our token drop model. Additionally, in Section 3.3, we propose to optionally replace the important tokens that have the lowest cumulative losses with unimportant tokens. Comparing token drop with token drop (half-rand) and token drop (rand) in Table 1, we see that adding randomness does not help. Finally, we see that the cumulative-loss-based dropping performs better than frequency-based dropping and random dropping. On how many tokens to drop. We report results with different token dropping percentages on training the BERT-base model in Table 4. We see that dropping more than 62.5% of the tokens yield worse results. By default, our experiments drop 50% of the tokens. On determining which layers to drop. Comparing token drop (half-rand) + stage-2 with token drop (layer rearranged) + stage-2, we can see that putting one full-sequence layer at the end of the model yields better results. On token dropping vs. token passing. Comparing token drop + stage-2 with token pass + stage-2, we see that passing the unimportant tokens instead of dropping them does not affect the performance. Recall that for layers where unimportant tokens are dropped, token dropping would make the input to such layers correspond to incoherent sentences, which could impact BERT's learning ability. However, we find that doing token passing makes pretraining slightly less efficient while providing no improvement on downstream performance. On token dropping vs. token averaging. Comparing token drop + stage-2 with token avg + stage-2, we see that average pooling instead of dropping unimportant tokens yields slightly worse results. This means that our importance-driven token selection is more efficient than directly averaging embedding across every nearby token pair. We test our method on BERT-base and BERT-large with a sequence length of 128 and 512. We report the results in Table 2. Overall, our proposed method performs similarly as the baseline method. As shown in Table 3, when taking the average across all GLUE and SQuAD scores and across all four settings (two BERT models times two sequence lengths) and two pretraining runs with different random seeds, our proposed token dropping method outperforms the baseline method by 0.3% (85.16% to 85.45%) in addition to the 25% pretraining time reduction. One strategy to improve data efficiency during language model pretraining is by designing better pretraining objectives (Lan et al., 2020; Clark et al., 2020; Raffel et al., 2020). Concurrently, researchers have also been exploring certain hardware properties to improve pretraining efficiency, e.g., mixed-precision training (Shoeybi et al., 2019) and huge-batch distributed training (You et al., 2020). Recently, Wu et al. (2021) propose to tackle the efficient pretraining problem through rare words or phrases, and they provide rare words with a note embedding to make models better aware of the contextual information in a sequence. The faster depth-adaptive transformer approach is applied to text classification tasks (Liu et al., 2021). It identifies important tokens by either computing the mutual information between each token and the given sequence label, or using a separate BERT model to exhaustively evaluate the masked language model loss for each token. There is a rich body of literature on faster inference of sequence generation problems, such as early layer exits during translation (Elbayad et al., 2020; Han et al., 2021), non-autoregressive machine translation (Gu et al., 2018; Tu et al., 2020b), and amortizing the cost of complex decoding objectives (Chen et al., 2018; Tu et al., 2020a; Pang et al., 2021a). Several ideas are particularly relevant to token-wise layer dropping: Zhang and He (2020) propose to use a fixed probability to drop an entire layer during pretraining; here, we use the more fine-grained 3781 token-wise layer dropping. The dynamic halting algorithm (Dehghani et al., 2019), motivated by the finding that transformers fail to generalize to many simple tasks, stops the processing of a token through upper layers if its representation is good enough. However, the implementation does not improve training time, as its goal is to improve performance. We present a simple yet effective approach to save BERT pretraining time. Our approach identifies unimportant tokens with practically no computational overhead and cuts unnecessary computation on these unimportant tokens for training. Experiments show that BERT models pretrained in this manner save 25% pretraining time, while generalizing similarly well on downstream tasks. We show that our token dropping approach performs better than average pooling along the sequence dimension. Future work will involve extending token dropping to pretraining transformer models that can process a much longer context, as well as extending this algorithm to a wider range of transformer-based tasks, including translation and text generation. The authors thank the anonymous reviewers for helpful feedback.
abstain objective method abstain method result abstain abstain abstain abstain other abstain objective abstain abstain abstain abstain method method result abstain abstain method method objective abstain abstain objective objective result abstain objective result other abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain method method method abstain method abstain method method method abstain abstain method abstain method abstain method method abstain method method abstain abstain abstain abstain method abstain abstain method method abstain abstain method method abstain abstain abstain method method method method method abstain other abstain abstain method method abstain method abstain abstain abstain method method method abstain method method method method abstain method method method method method abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain result result abstain result result abstain abstain result abstain result abstain result abstain result abstain abstain result abstain abstain other other other other other other other abstain other other method method abstain result abstain other