In this paper, we propose SkipBERT to accelerate BERT inference by skipping the computation of shallow layers. To achieve this, our approach encodes small text chunks into independent representations, which are then materialized to approximate the shallow representation of BERT. Since the use of such approximation is inexpensive compared with transformer calculations, we leverage it to replace the shallow layers of BERT to skip their runtime overhead. With off-the-shelf early exit mechanisms, we also skip redundant computation from the highest few layers to further improve inference efficiency. Results on GLUE show that our approach can reduce latency by 65% without sacrificing performance. By using only two-layer transformer calculations, we can still maintain 95% accuracy of BERT. 1 1 Introduction Pre-trained language models, such as ELMo (Pe-ters et al., 2018), GPT (Radford et al., 2018), BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), and RoBERTa (Liu et al., 2019), have yielded significant improvements to NLP tasks. Despite the gain in accuracy, these models have significant demands in computation and inference time, limiting their use in resource-constrained or latency-sensitive applications. Therefore, it is desirable to reduce the computational overhead of these models while retaining acceptable accuracy. Knowledge distillation (KD, Hinton et al. 2015) facilitates the transfer of knowledge embedded in pre-trained language models into small student models (Sanh et al., 2019; Sun et al., 2019; Jiao et al., 2020), which usually reduces the redundant parameters of BERT in a uniform manner. Early exit mechanisms (Xin et al., 2020; Zhou et al.,  Corresponding author 1 Source code is available at https://github.com/ LorrinWWW/SkipBERT . 2020; Liu et al., 2020) then use an adaptive number of transformer layers during inference, aiming to reduce redundant calculations from the highest few layers. However, since they build the sequence representation from scratch for each forward pass, they require a certain number of lower layers to capture basic syntactic and semantic information, making it difficult to further reduce inference costs. This naturally raises a question: Can we reduce the computation at the lower transformer layers? In this paper, we propose SkipBERT, a novel scheme that skips the computation at the shallow transformer layers of BERT. As revealed by Jawa-har et al. (2019); Rogers et al. (2020), the lower layers of BERT mainly focus on short-distance context, while the higher layers are able to capture long-range dependencies. Therefore, it is reasonable to assume that, at lower layers, even if distant tokens are masked, the representation for each token will not vary dramatically. Here, by sweeping over the input text, we get short chunks (n-grams) and use their representations to approximate the hidden states of BERT's lower layers. We then precompute and store representations of text chunks in a precomputed lookup table (PLOT). Thus, during inference we only need to access PLOT to get the representations of short chunks, which is inexpen-7287 sive compared with transformer computation. Fig. 1 compares the inference procedure between vanilla BERT and our proposed SkipBERT. In BERT, the input text needs to be processed by a large number of transformer layers in turn, leading to high latency in inference. In comparison, SkipBERT precomputes the hidden states of lower transformer layers, which are accessed via table lookups, rather than computed in inference-time. Moreover, SkipBERT exhibits effective compatibility with early exit mechanisms: Since the initial sequence representation in our work is partially contextualized (thanks to PLOT) rather than individual word embeddings, SkipBERT allows exiting from a relatively earlier layer than typical BERT variants, while maintaining good accuracy. We empirically verify this in Section 4.5. Therefore, our approach can skip the calculations of lower and higher layers for the same input, thereby further improving the inference speed. We present SkipBERT to avoid computation at BERT's lower layers during inference. Instead, we construct PLOT and use it to approximate their hidden states. We incorporate early exit mechanisms as an enhancement to skip redundant computation, leading to further network acceleration. We conduct extensive experiments on GLUE. Compared with BERT, SkipBERT is capable of accelerating inference by up to 65% without compromising GLUE score, or accelerating by 82% while retaining 95% accuracy. Knowledge Distillation (Hinton et al., 2015) provides an effective way to transfer the knowledge embedded in a teacher network to a student network. The student network is usually more lightweight than the teacher network and thus more computationally efficient. The student network can be structurally identical to the teacher but contains fewer layers or hidden units, e.g. BERT-PKD (Sun et al., 2019), DistilBERT (Sanh et al., 2019), TinyBERT (Jiao et al., 2020), MiniLM (Wang et al., 2020), and BERT-EMD (Li et al., 2020). Meanwhile, some work adopts specifically designed networks, e.g. SqueezeBERT (Iandola et al., 2020) and MobileBERT (Sun et al., 2020), to reduce the computation per layer. Input-adaptive inference allows models to choose different computational paths according to the input during inference. In this way, simpler input samples usually require less calculation to make predictions. Recently, DeeBERT (Xin et al., 2020) adapts confidence-based BranchyNet (Teer-apittayanon et al., 2016), which uses entropy as an early-exit criterion. FastBERT (Liu et al., 2020) uses self-distillation to train the branch classifiers. RightTool (Schwartz et al., 2020) leverages the same early-exit criterion as in the Shallow-Deep Network (Kaya et al., 2019), i.e., softmax scores of predictions. PABEE (Zhou et al., 2020) stops inference when the intermediate predictions of the internal classifiers remain unchanged consecutively. Precomputation has also been studied in information retrieval, where documents are assumed to be stored at local database so their representation can be precomputed (Gao et al., 2020). However, this method may not be suitable for other tasks where the input text is unknown before inference. During training, SkipBERT consists of two groups of transformer layers, local transformer layers for encoding short-distance context, and global transformer layers for leveraging the full context. Once pre-training finishes, our approach will replace local transformer layers with PLOT, which stores the hidden states of local transformer layers; we also enhance global transformer layers with early exit mechanisms to further accelerate inference speed. Fig. 2 presents the overview of our system. As shown in Fig. 3, we sweep over the input text to get three-token chunks (tri-grams, X i = [ x i  1 , x i , x i +1 ] ), which will also be taken as the index entries of PLOT later. We let cross-border tokens be padding tokens, i.e. x  1 = x n = x [PAD] . We show in Section 4.8.2 that using longer text chunks (e.g. 5-grams) will improve accuracy since they can bring more context information than trigrams. However, the number of 5-grams is too large to be enumerated and stored, and thus it is hard to use them in actual applications. Fig. 3 illustrates our procedure to leverage local context. By mapping each word to a d -dimensional embedding, we denote the chunk embeddings by  X i  R 3  d . For local transformer layers, we inject position embeddings P  R 3  d , and define the initial chunk representations as follows: H (0) i = LN (  X i + P ) (1) where LN (  ) is layer normalization. We use L loc transformer layers to leverage the local context of each text chunk. For layer 0  m < L loc , we have: H ( m +1) i = Transformer ( m ) ( H ( m ) i ) . Note that since each chunk is short enough, it would be possible to precompute these representations before inference. More importantly, these representations are good approximations of those produced by the respective shallow layers of BERT. Thus, given a tri-gram, the embedding produced from ( L loc  1) -th layer is taken as its respective data entry stored in PLOT. We also precompute bi-grams and uni-grams following the same procedure of tri-grams. When a lookup of tri-gram fails (out-of-vocabulary, OOV), the system will resort to bi-grams or uni-grams as an alternative. Specifically, we randomly replace  % of trigrams by bi-grams during training. On the one hand, such random replacement allows the model to encounter bi-grams during training, so as to better handle OOVs in inference; on the other hand, it can also be considered a variant of Dropout (Srivas-tava et al., 2014), which drops tokens rather than Transformer Layer Transformer Layer Transformer Layer Transformer Layer Transformer Layer Transformer Layer Transformer Layer Transformer Layer Transformer Layer ... ... hidden units, thereby improving the robustness of our approach. Section 4.7 shows  = 10% works well with different OOV rates. We also show in Section 4.8.2 that even bi-grams have a clear advantage over the baseline, which can be seen as an extreme case when all tri-gram lookups fail. Now we get a list of contextualized chunk embeddings. Here we aggregate them to form a feature sequence corresponding to the original input text. Each token occurs at three consecutive tri-grams, as shown in Fig. 3. By calculating a weighted sum of embeddings that correspond to the same token, we can leverage its context of five tokens:  h i = (cid:88) j =  1 , 0 , 1 H ( L loc ) i + j, 1  j  Gate ( H ( L loc ) i + j, 1  j ) , (3) where Gate (  ) is a sigmoid-based gating mechanism such that Gate ( x ) =  ( v G  x + b G ) , where v G is a learnable vector and b G is a learnable scalar. Note that these embeddings do not have a sense of the order of the sequence. So we need to inject position and segment embeddings before sending them to the subsequent transformer layers: h (0) i = (cid:40) LN ( h i +  p i +  s A ) , if x i  A, LN ( h i +  p i +  s B ) , if x i  B, (4) where  p i and  s A/B are position and segment embeddings respectively as in Devlin et al. (2019). We denote by h (0) = [ h (0) i ] 0  i<n the aggregated sequence representation. We use L glo transformer layers to further contextualize it. For layer 0  m < L glo , we have: h ( m +1) = Transformer ( m + L loc ) ( h ( m ) ) . Since we focus on text classification and regression tasks, we use the representation corresponding to token x [CLS] to compute logit scores: z = Classifier ( h ( L glo ) [CLS] ) (6) where Classifier (  ) is a two-layer feedforward neural network. When an early exit mechanism is activated, we compute logit scores for each global transformer layer as follows: z ( m ) = Classifier ( m ) ( h ( m ) [CLS] ) (7) We adopt a simple confidence-based early exit mechanism, i.e., once the prediction's maximum logit score is higher than a pre-defined threshold, the result will be returned without passing through the next transformer layers. We mainly adopt the two-stage learning procedure proposed in TinyBERT (Jiao et al., 2020). It includes general distillation (GD) conducted on large-scale unlabeled corpora, and task-specific distillation (TD) to learn from fine-tuned BERT. General Distillation We perform distillation on the hidden states and attention scores. We compute loss on the chunk aggregation layer and global transformer layer. The local transformer layers are trained with supervision signals from upper layers. The loss is defined as follows: LGD = L att + L hid (8) and we define L att and L hid as the mean-squared error (MSE) of attention scores and hidden states between the teacher (T) and student (S): L att = L glo (cid:88) m =1 MSE ( a ( m ) S , a ( g att ( m )) T ) (9) L hid = L glo (cid:88) m =0 MSE ( h ( m ) SW hid , h ( g hid ( m )) T ) (10) where a ( m +1) and h ( m +1) represent the attention score matrix and hidden states of the m -th transformer layer; h (0) is the outputs of chunk aggregation layer; W hid is a learnable matrix to transform the hidden states of the student into the same space as the teacher; g att (  ) and g hid (  ) define the layer mapping function between the student and teacher. For attention-based distillation, we use the uniform mapping strategy to leverage the heterogeneous attention patterns across different layers. For hidden states-based distillation, we use top mapping strategy since the initial sequence representation (outputs of chunk aggregation) are already partially contextualized. The detailed illustration of layer mapping is presented at Appendix E. Task-Specific Distillation We start from the generally distilled SkipBERT, and use fine-tuned BERT as the teacher for task-specific distillation. The loss is defined as follows: LTD =  ( L att + L hid ) + L pred (11) where  is a factor to control the loss weight; L pred is the prediction loss that will be defined below. For classification, the loss function L pred is calculated via cross entropy: L pred = CE ( z S /, z T / ) (12) where z S are the logits predicted by the student; z T are the logits predicted by the teacher;  is the temperature to smooth the probability distribution to facilitate distillation training. For regression, the loss is instead calculated by MSE, i.e., L pred = MSE ( z S , z T ) . Early Exit Specifically, when SkipBERT enables early exit mechanisms, we need to train internal classifiers to predict based on the hidden states of their respective layers. Overall, we train the model to minimize a weighted average loss as follows: L pred = (cid:80) L glo m =1 m L ( m ) pred (cid:80) L glo m =1 m (13) where L ( m ) pred is the loss between the predictions of the teacher and the m -th intermediate classifier of the student. Considering that the local transformer layers mostly capture generalized knowledge, which do not vary significantly across different tasks, we do not update the local transformer layers during fine-tuning. Therefore, once general distillation is finished, we can compute their hidden states to construct PLOT. To ensure fast response, PLOT should ideally be loaded in the server's RAM during inference. However, such a table could be too large to fully fit into 7290 Model MACs Latency GLUE CoLA SST-2 MRPC STS-B QQP MNLI QNLI RTE WNLI Score (8.5k) (67k) (3.5k) (5.7k) (364k) (393k) (108k) (2.5k) (0.6k) BERT 12 10.9G 100% 78.3 52.1 93.5 88.9/84.8 87.1/85.8 71.2/89.2 84.6/83.4 90.5 66.4 65.1 BERT 6 -PKD 5.4G -49% -43.5 92.0 85.0 -/81.6 70.7/-81.5/81.0 89.0 65.5 DistilBERT 6 5.4G -49% -49.0 92.5 86.9 -/81.3 70.1/-82.6/81.3 88.9 58.4 BERT-of-Theseus 6 5.4G -49% 77.1 47.8 92.2 87.6/83.2 85.6/84.1 71.6/89.3 82.4/82.1 89.6 66.2 65.1 TinyBERT 6 v2 5.4G -49% -46.1 92.6 88.0/--/83.9 71.3/-84.4/83.1 89.8 69.7 BERT-EMD 6 5.4G -49% 78.7 47.5 93.3 89.8/86.4 87.6/86.8 72.0/89.3 84.7 /83.5 90.7 71.7 65.1 SkipBERT 6+6 5.4G -49% 78.9 52.7 93.3 88.9/85.0 87.0/85.8 71.9/89.2 84.3/ 84.2 90.6 70.6 65.1 w/ exit 3.7G -65% 78.3 50.8 91.9 88.6/84.8 86.7/85.4 71.8/89.0 83.8/83.8 90.2 69.8 65.1 BERT mini4 0.4G -66% 65.8 0.0 85.9 81.1/71.8 75.4/73.3 66.4/86.2 74.8/74.3 84.1 57.9 62.3 BERT small4 1.6G -66% 71.2 27.8 89.7 83.4/76.2 78.8/77.0 68.1/87.0 77.6/77.0 86.4 61.8 62.3 DistilBERT 4 3.6G -66% -32.8 91.4 82.4/--/76.1 68.5/-78.9/78.0 85.2 54.1 BERT 4 -PKD 3.6G -66% -24.8 89.4 82.6/--/79.8 70.2/-79.9/79.3 85.1 62.3 TinyBERT 4 v2 0.6G -66% -25.3 90.0 85.4/--/80.4 68.9/-81.2 80.3 86.2 63.9 BERT-EMD 4 0.6G -66% 73.6 25.6 91.0 87.6/82.4 83.6/82.3 69.3/87.9 82.1 /80.6 87.2 66.2 65.1 SkipBERT 6+4 0.6G -66% 75.6 39.8 91.3 87.7 /82.7 84.1/82.8 70.4/88.3 82.0/ 81.6 88.5 66.1 65.1 w/ exit 0.5G -74% 75.1 36.0 91.2 87.6/ 83.5 84.1/82.8 70.4/88.3 81.9/81.3 88.5 64.8 65.1 SkipBERT 6+2 1.8G -82% 74.0 36.0 90.9 85.9/80.5 82.0/80.6 70.2/88.6 80.2/79.9 86.6 63.6 65.1 Table 1: Results on the GLUE benchmark. RAM. Hence we propose to adopt memory-mapped files ( mmap ), which allows for file access via the virtual memory mechanism. By using mmap , the frequently used chunk embeddings reside in RAM for fast lookup, while the rare chunks can be stored on SSD, and will be loaded to RAM only when the system demand-pages them. Appendix D presents a simple implementation of PLOT. We use the corpora of Wikipedia 2 and BooksCor-pus 3 (Zhu et al., 2015) to perform general distillation. For task-specific distillation, we mainly evaluate SkipBERT and compare it with other baselines on the GLUE benchmark (Wang et al., 2018). Appendix F provides some details. We denote by SkipBERT 6+6 the scheme with 6 local transformer layers (converted to PLOT) and 6 global transformer layers, each having a hidden size of 768 and intermediate size of 3072. For direct comparisons with 4-layer baselines, we instantiate SkipBERT 6+4 with 4 thin global transformer layers (hidden size of 312 and intermediate size of 1200). We also instantiate SkipBERT 6+2 with only 2 https://dumps.wikimedia.org/ 3 https://yknzhu.wixsite.com/mbweb 2 global transformer layers to further reduce the latency. Training For general distillation, we randomly initialize SkipBERT, and pre-train it with Lamb optimizer (You et al., 2019). We use linear learning rate decay with the peak learning rate of 1e-3 and a batch size of 2048 for around 80k steps, including 4000 warm-up steps. For task-specific distillation, under the supervision of a fine-tuned BERT, we use AdamW (Kingma and Ba, 2015) to train 20 epochs with a learning rate of 2e-5. We slightly tune the hyper-parameters across different tasks, and the details can be found in Appendix B. We do not use any data augmentation strategies. Inference Following prior work, we evaluate latency by performing inference on a per-instance basis, i.e. the batch size for inference is set to 1. This is a common latency-sensitive scenario when processing individual requests from different users. We note that latency on modern GPUs is not sensitive to the hidden size, but mainly depends on the number of sequential operations, i.e. the number of network layers. We report the median performance over 5 runs with different random seeds. We submitted our model predictions to the official GLUE evaluation server 4 to obtain results on the test set, as summarized in Table 1. We present the results of TinyBERT v2 reported by Li et al. (2020) as the v2 model employs more training corpora than v1, and they eliminate the data augmentation strategy for a fair comparison. By comparing with baselines (we compare with 6-layer models and 4-layer models separately), we can see that SkipBERT outperforms all compared approaches in terms of GLUE score. Compared with TinyBERT, as we mainly follow their distillation process, our approach shows clear advantages on all tasks. BERT-EMD employs a more sophisticated task-specific distillation process based on general-distilled TinyBERT, and further improves the overall performance. Nevertheless, SkipBERT still maintains an advantage in the overall score. Specifically, SkipBERT 6+4 has a similar inference speed to the 4-layer baselines, but achieves higher accuracy on most tasks. We consider that a 4-layer model is somewhat too shallow to capture complex dependencies from scratch. In contrast, SkipBERT effectively compensates by adding more layers in effect, even though their computation is skipped by PLOT search during inference. These layers are useful to capture the basic linguistic information, thereby reducing the burden on subsequent layers. Moreover, our method can further reduce the latency with only a slight loss in accuracy. SkipBERT 6+2 which performs only two-layer transformer calculations maintains accuracy comparable to 4-layer models. For the 6-layer track, TinyBERT and BERT-EMD both achieve performance comparable to the teacher model. However, SkipBERT 6+6 also shows competitive results, especially for the challenging CoLA task (predicting linguistic acceptability judg-ments), on which previous methods do not work well. The local transformer layers of SkipBERT can effectively capture the short-distance grammatical knowledge, e.g. subject-verb-object word order and verbal argument structure, etc., which is considered crucial to CoLA (Warstadt et al., 2019). The early exit mechanism, tagged by w/ exit in Table 1, provides a flexible way to tune the speed-accuracy tradeoff. With early exit enabled, both SkipBERT 6+6 and SkipBERT 6+4 achieve further improvements on inference speed with only a mi-nor decrease in accuracy. More exploration will be done in Section 4.5. We also investigate the effectiveness of SkipBERT on the reading comprehension task, SQuAD v1.1 (Rajpurkar et al., 2016a). Following previous work, we treat this task as sequence labeling and predict the possibility of each token as the start or end of answer span. Table 2 shows that SkipBERT outperforms all the baselines with large margins. This experiment shows that our approach also works well for relatively complicated task forms. Here we investigate the compatibility between early exit mechanisms and SkipBERT. We draw the accuracy-latency curve by tuning the early exit threshold. The goal is to enlarge the area under the curve  so the model can maintain good accuracy when the average exit layer is small. Fig. 4 compares the results of TinyBERT 4 v2 and SkipBERT 6+4 , both using the same early exit mech-7292 Operation MACs Latency d = 768 / 312 d = 768 / 312 Retrieve Text Chunks 155 s / 114 s Chunks to IDs -31.4 s Retrieve from mmap * -22.8 s / 18.4 s Send to GPU 101 s / 64.1 s Aggregate Text Chunks -48.4 s / 42.7 s Each Transformer Layer 906M / 146M 1.22 ms / 1.18 ms Each Classifier 591K / 98K 116 s / 112 s Table 3: Breakdown of computation and average latency of text with 128 tokens (including padding to-kens). anism. We observe that SkipBERT consistently outperforms TinyBERT on both MRPC and SST-2. Specifically, the curve of SkipBERT is flatter than that of TinyBERT, which indicates that even if SkipBERT is forced to exit at a relatively shallow layer, it can still maintain a desirable accuracy. Compared with baselines, our approach starts inference based on PLOT search results rather than from scratch, so even at a lower layer, the representation is well-learned for making predictions. Table 3 presents the breakdown of computation and average latency of SkipBERT. Detailed hardware information can be found at Appendix A. We can observe that the transformer layers account for the majority of inference time. We note that there may be some variation in the latency of retrieving data from mmap , depending on the cache memory managed by the operating system. Fig. 5 presents the latency distribution of retrieving chunks contained in a text sequence with OOV: 0% 11.2% 14.5% 24.7% PLOT size: 168G 59.2G 12.1G  = 0% 88.0/91.4 87.0/90.6 86.3/90.1 85.8/89.6  = 5% 88.2/91.4 88.2/91.7 88.0/91.3 87.3/91.0  = 10% 88.2/91.6 88.0/91.4 88.0/91.4 88.0/91.2  = 20% 88.0/91.4 87.7/91.2 88.0/91.3 88.0/91.2 Table 4: Influence of space costs on the model accuracy. 128 tokens under different RAM sizes. We perform experiments in Docker containers to limit the RAM size; more results can be found in Appendix G. The upper half of Fig. 5 shows that with enough RAM, the system can directly collect chunk embeddings from RAM, yielding latencies clustered around 20 s. Meanwhile, with a smaller RAM as shown in the lower half of Fig. 5, most of the latency is still around 20 s but a small portion of items take several hundred s due to cache misses. We also observe that the long tail of latency is distributed in several clusters, mainly due to I/O queuing. However, even under heavy I/O load, retrieving data from mmap takes less time than the computation of a single transformer layer. The previous sections prioritize accuracy and ef-ficiency by sacrificing space. Reducing the space costs (by dropping less frequent chunks) allows users to use more economical hardware, but it will lead to OOV issues which may compromise accuracy. Here we only count OOV for tri-grams, since OOVs for bi-grams rarely occur ( < 0.5%) and have little impact on the final performance. We collect tri-grams on news corpora 5 and training sets of GLUE to construct PLOT. Table 4 shows results by reducing the space costs. = 0% means that the model does not see any bi-grams during training. In this case, if the model encounters a tri-gram lookup failure and reverts to bi-grams, the performance will suffer to some extent. When we randomly replace  % of tri-grams with bi-grams during training, the model becomes more robust to OOVs, and can even slightly improve accuracy. We find  = 10% works well for all cases, and thus we use it as the default value. tage even if the OOV rate is at a moderate level. As we will see later at Section 4.8.2, if we only use bi-gram embeddings, i.e. the OOV rate is 100%, our approach is still better than the baseline that does not apply PLOT. To understand why the backoff strategy, namely to replace tri-grams with bi-grams for OOVs, does not hurt accuracy, we investigate the similarity between them. As shown in Fig. 6, most of them are similar, confirming the feasibility of our backoff strategy; but there is also a long tail where bi-grams cannot well compensate for the missing tri-grams. Fig. 6 also shows some examples with different similarities. Generally, auxiliary tokens that do not contain much meaning by themselves tend to rely more on context. Meanwhile, tokens rich in semantics, e.g. noun phrases, do not vary much in embedding under different ranges of context. We conduct an ablation study in this subsection. We only pre-train SkipBERT on the Wikipedia corpus for 1 epoch for fast validation. We also prepare a generally distilled small BERT 4 (the model architecture is identical to TinyBERT 4 ) with the same setup and corpus as a baseline. We report the results on the development set. Table 5 compares the results with different num-bers of local transformer layers. BERT 4 is a base-Model r.f. CoLA MRPC MNLI BERT 4 -23.7 85.0/89.6 79.8/79.6 SkipBERT 6+4 w/ 1-gram 1 23.5 85.3/89.5 79.8/79.5 w/ 2-gram 3 29.1 86.3/90.1 80.8/81.0 w/ 3-gram ctr. line that does not employ any skipping mechanism. We can see that all settings that use additional local transformer layers have better performance than BERT 4 , indicating the effectiveness of our approach. In general, the performance increases when we gradually enlarge the number of local transformer layers. CoLA benefits most from the local transformer layers due to better modeling of short-distance context. However, when it reaches a certain number of local transformer layers, the improvement becomes minimal. We believe that since each token only has the context of five tokens, too many layers may increase the risk of overfitting, which harms the performance. Thus we adopt 6 layers as our default setting. We also construct variants that replace local transformer layers with a single-layer FFN or CNN, which are computationally lightweight and thus may not need precomputation. However, their accuracy improvement against BERT 4 is very limited, which shows that even for short-distance context, using a relatively complex and deep network is beneficial to the final performance. We investigate the effect of short-distance context leveraged in the local transformer layers of SkipBERT. Table 6 presents the comparisons of using different ranges of short-context in local transformer layers. 1-grams are equivalent to conventional word embeddings, and the performance is similar to the baseline. When using 2-grams, SkipBERT obtains notable improvements since each token can now access its direct neighbors in local transformer layers. 3-grams and 5-grams bring consistent improvements to all tasks. Generally, the results are improved when we broaden the receptive field of local transformer layers, showing that 7294 more contexts are always beneficial. However, due to its large number, it would be hard to enumerate n-grams with n > 3 . It might require certain pruning or compression strategies, which we leave as future work. In addition, we also study the effect of the weighted sum used in chunk aggregation, Eq. (3). We add comparison against a variant that only uses the embedding of the central token of each chunk, denoted ctr. only. Table 6 shows that the weighted sum brings improvements over all tasks for the 3-gram setting. However, it is not as effective for the 5-gram setting on MRPC and MNLI. We believe using a weighted sum for five chunks may confuse important semantics and thus affect the accuracy; while for 3-gram setting, this problem is not as serious, and using a weighted sum for neighbor chunks can bring more context information to improve the accuracy. 4.8.3 Effect of Distillation Objective Model CoLA MRPC MNLI SkipBERT 6+4 32.9 86.3/90.4 80.9/81.1 w/o GD-att 29.9 81.9/87.2 81.3/81.6 w/o GD-hid 28.7 85.5/90.0 80.9/81.1 w/o TD-att 31.3 85.8/89.6 80.1/80.1 w/o TD-hid 30.5 85.8/89.9 80.4/80.1 Table 7: Effect of different distillation objectives. We here show the effects of different distillation objectives. We try to eliminate attention-based or hidden state-based distillation. Results in Table 7 indicate that all distillation objectives are helpful both in the general distillation and task-specific distillation process. In general distillation, both attention and hidden states-based distillation are critical to the final performance of relatively small datasets, e.g. CoLA and MRPC. But for large-scale datasets, e.g. MNLI, removing attention based distillation even improves the performance, which may imply that the student model can benefit more from a fine-tuned teacher model as long as the downstream task has enough data. In the task-specific distillation, the two distillation objectives are marginally helpful for CoLA and MRPC, while acting more importantly for MNLI. The original TinyBERT uses a data augmentation strategy for all tasks during fine-tuning, which significantly enlarges the training set and makes the effect of task-specific distillation more significant. In this paper, we proposed SkipBERT, a straightforward yet effective approach to skip the computation of BERT's shallow layers. We used representations of short text chunks to approximate BERT's shallow representation, and stored them in PLOT for fast retrieval during inference. Empirical results showed that SkipBERT could achieve performance comparable to BERT while significantly reducing inference time. In the future, we would like to leverage discontinuous text chunks to further improve the accuracy and inference speed. We will also try to reduce storage requirements with appropriate pruning and compression strategies. This work was supported by the Key Research and Development Program of Zhejiang Province of China (No. 2021C01009), NSF of China Grant No. 62050099, and the Fundamental Research Funds for the Central Universities.
objective result method result result method abstain abstain abstain abstain other abstain abstain method objective abstain abstain method abstain method objective abstain abstain abstain result result method method abstain method objective other other other other other other other other other other other other abstain method method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain other abstain other abstain method abstain method abstain other abstain abstain method method abstain method method method method abstain abstain abstain method method method method abstain other abstain abstain abstain abstain abstain abstain abstain method method other method method method method method abstain method method result abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other abstain result abstain result abstain result abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain result method other