Transferring the knowledge to a small model through distillation has raised great interest in recent years. Prevailing methods transfer the knowledge derived from mono-granularity language units (e.g., token-level or sample-level), which is not enough to represent the rich semantics of a text and may lose some vital knowledge. Besides, these methods form the knowledge as individual representations or their simple dependencies, neglecting abundant structural relations among intermediate representations. To overcome the problems, we present a novel knowledge distillation framework that gathers intermediate representations from multiple semantic granularities (e.g., tokens, spans and samples) and forms the knowledge as more sophisticated structural relations specified as the pair-wise interactions and the triplet-wise geometric angles based on multi-granularity representations. Moreover, we propose distilling the well-organized multi-granularity structural knowledge to the student hierarchically across layers. Experimental results on GLUE benchmark demonstrate that our method outperforms advanced distillation methods. Recent years have witnessed a surge of pre-trained language models (Devlin et al., 2019; Lewis et al., 2020; Clark et al., 2020; Brown et al., 2020). Building upon the transformer architecture (Vaswani et al., 2017) and pre-trained on large-scale corpora using self-supervised objectives, these PLMs have achieved remarkable success in a wide range of natural language understanding and generation tasks. Despite their high performance, these PLMs usually suffer from high computation and memory costs, which hinders them from being deployed  Corresponding authors: Chongyang Tao and Dongyan Zhao. and embedded devices. Various attempts have been made to compress the huge PLMs into small ones with minimum performance degradation. As one of the main approaches, knowledge distillation (Hinton et al., 2015) utilizes a large and powerful teacher model to transfer the knowledge to a small student model. Based on the teacher-student framework, Jiao et al. (2020); Wang et al. (2020) distilled the token-level representations and attention dependencies to the student, Sanh et al. (2019); Sun et al. (2019) taught the student to mimic the output logits of the teacher, Sun et al. (2020) enforced the student's representation to be closed to the teacher's while pushing negative samples to be far apart. Although proved effective, existing approaches have some flaws. For one thing, these distillation methods only adopted the representations of mono-granularity language units (i.e., token-level or sample-level), while neglecting other granularity. For another, their distillation objectives either matched the corresponding representations between the teacher and the student or aligned the attention dependencies, failing to capture more sophisticated structural relations between the representations. To address these issues, in this paper we propose a novel knowledge distillation framework named M ultiG ranularity S tructural K nowledge D istillation (MGSKD) through answering the three research questions: (1) which granularity should the knowledge be, (2) what form of knowledge is effective to transfer and (3) how to teach the student using the knowledge. For the  which  question, given that natural languages have multiple semantic granularities, we consider the intermediate representations in three granularities: tokens, spans and samples. Specifically, we first take the sub-word tokens as the smallest granularity, then 1001 select phrases and whole words as spans for they hold complete meanings, and finally treat the whole input texts as samples. We use mean-pooling to obtain the representations of spans and samples based on token representations. For the  what  question, we propose to leverage the sophisticated structural relations between the representations as the knowledge. Concretely, instead of aligning the corresponding representations of the teacher and the student, we propose to form the knowledge as the pair-wise interactions and the triplet-wise geometric angels of a group of representations. For the  how  question, following the recent findings that the bottom layers capture syntactic features while the upper layers encode semantic features (Jawahar et al., 2019), we conduct hierarchical distillation where the bottom layers of the student are taught token-level and span-level knowledge while the upper layers learn sample-level knowledge. We conduct comprehensive experiments on standard language understanding benchmark GLUE (Wang et al., 2018). Experimental results demonstrate that our knowledge distillation framework outperforms strong baselines methods. Surprisingly, MGSKD achieves comparable or better performance than BERT base on most of the tasks on GLUE, while keeping much smaller and faster. Our contributions in this paper are three folds:  We are the first to leverage multi-granularity semantic representations in language (i.e., the representations of tokens, spans and samples) for knowledge distillation. We propose to form the knowledge as sophisticated structural relations specified as the pair-wise interactions and the triplet-wise geometric angles based on multi-granularity representations. Language Model Compression. Pre-trained language models (Devlin et al., 2019; Clark et al., 2020; Brown et al., 2020) perform remarkably well on various applications but at the cost of high computation and memory usage. To deploy these powerful models into resource-scarce scenarios, various attempts have been made to compress the language models into small ones. Quantization methods (Zafrir et al., 2019; Shen et al., 2020; Zhang et al., 2020; Bai et al., 2021) convert the model parameters to lower precision. Pruning approaches identify then remove unimportant individual weights or structures (Michel et al., 2019; Fan et al., 2019; Gordon et al., 2020; Hou et al., 2020). Weight sharing techniques (Dehghani et al., 2018; Lan et al., 2019) allow the model to reuse the transformer layer multiple times to reduce parameters. Knowledge Distillation. Knowledge distillation (Hinton et al., 2015) is another major line of research to do model compression, which is the main concentration in this paper. Hinton et al. (2015) first proposed to minimize the KL-divergence between the predicted distributions of the teacher and the student. Sanh et al. (2019); Sun et al. (2019); Liang et al. (2020) adopted this objective to teach the student on masked language modeling or text classification tasks. Romero et al. (2014) proposed to directly match the feature activations of the teacher and the student. Jiao et al. (2020) followed the idea and took the intermediate representations in each transformer layer of the teacher as one of the knowledge to be transferred. Tian et al. (2019) proposed a contrastive distillation framework where the teacher's representations were treated as positives to the corresponding student's representations. Sun et al. (2020); Fu et al. (2021) customized this idea to language model compression and proved its effectiveness. Researchers also attempted to use the mutual relations of representations as the knowledge to transfer. In the literature of image classification, Peng et al. (2019); Tung and Mori (2019); Park et al. (2019) pointed out that the relations of the image representations of the teacher should be preserved in the student's feature space, and adopted a series of geometric measurements to model the sample relations. For distilling transformer models, Park et al. (2021) enforced the relations across tokens and layers between the teacher and the student to be consistent. Jiao et al. (2020); Wang et al. (2020, 2021) used the attention dependencies between tokens to teach the student. In this paper, we propose to transfer the multi-granularity knowledge to the student. Different from previous works that only considered a single granularity of representations, we jointly transfer the token-level, span-level and sample-level structural knowledge. And compared with Shao and Chen (2021) which considered the multi-granularity visual features in an image as the knowledge, our method works in a different modality, presents a different definition of granularity, 1002 polic-eman Jackie Chan acted as a in a series of action movies . and prepares the multi-granularity knowledge as the structural relations among representations. We propose M ultiG ranularity S tructural K nowledge D istillation, a novel framework to distill the knowledge from a large transformer language model to a small one. Different from previous works that transferred the knowledge derived from either token-level or sample-level outputs, we prepare the knowledge in three semantic granularities: token-level, span-level and sample-level. Given some granularity of representations of the teacher model, we form the knowledge as the structural relations, i.e., the pair-wise interactions and the triplet-wise geometric angles, between the representations. We then distill the well-organized structural knowledge to the student hierarchically across layers, where the token-level and the span-level knowledge are transferred to the bottom layers to provide more syntactic guidance while the sample-level knowledge is transferred to the upper layers to offer more help of semantic understanding. The framework of MGSKD is illustrated in Figure 1. Natural languages have multiple granularities of conceptual units. In the context of pre-trained transformers (Devlin et al., 2019), the basic unit is the tokens produced by sub-word tokenizers (Wu et al., 2016; Radford et al., 2019). Several consecutive tokens become a text span, and the sample is comprised of all the tokens it contains. Existing knowledge distillation approaches (Jiao et al., 2020; Wang et al., 2020; Sun et al., 2020; Fu et al., 2021) focused on one granularity of representation, neglecting that texts are built upon language units from multiple granularities. Intuitively, incorporating multi-granularity representations in knowledge distillation may provide more guidance since the student can be taught how to compose the semantic concepts from small granularities to larger ones. Therefore, we propose to gather multi-granularity representations for knowledge distillation. We construct three granularities of representations: tokens, spans that hold complete meanings, and samples. Token Representation. The first granularity is the sub-word token, which is the foundation of high-level granularity. Given an input text, a tokenizer such as WordPiece (Wu et al., 2016) splits it into n tokens x = [ t 1 , t 2 , . . . , t n ] . The tokens are converted to a sequence of continuous representations E = [ e 1 , e 2 , . . . , e n ]  R n  d through the embedding layer. For the sake of clarity, we treat the embedding layer as the 0 -th layer and set H 0 = E . Then the token embeddings H 0 are passed to L stacked transformer layers. The l -th layer takes the output representations H l  1 of the previous layer as its input, and returns the updated representations H l using multi-head attention (MHA) and position-wise feed-forward network (FFN). Herein, we obtain L +1 layers of token 1003 representations { H l } Ll =0 where H l  R n  d . Span Representation. The second granularity is the span, which is comprised of several consecutive tokens. Different from SpanBERT (Joshi et al., 2020) that randomly selects token spans whose start positions and lengths are sampled from some distributions for masked language modeling, we propose to extract spans that have complete meanings. Widely adopted sub-word tokenizers in pre-trained transformers split some of the English words into several sub-word tokens. We consider these whole words consisting of multiple sub-word tokens, and phrases, as meaningful spans. Sub-word tokens for whole words are easy to obtain using WordPiece tokenizer (Wu et al., 2016). While for phrase identification, we train a classifier-based English chunker on CoNLL-2000 corpus (Tjong Kim Sang and Buchholz, 2000) following the instructions 1 . We then use the trained chunker to extract noun phrases (NP), verb phrases (VP), and prepositional phrases (PP). These identified phrases are tokenized by WordPiece tokenizer to obtain tokens. Herein, we can obtain n s token spans x span = [ s 1 , s 2 , . . . , s n s ] , where s i = [ t j , t j +1 , . . . , t j + n si  1 ] denotes the i th span that starts at the j -th token and contains n s i tokens. We then build span representations based on token representations using mean pooling:  h li = Pool ( H lj : j + n si ) , (1) where  h li  R d is the representation of the i -th span in layer l . We obtain L + 1 layers of span representations as {  H l } Ll =0 where  H l  R n s  d . Sample Representation. The third granularity is the input text sample itself. Based on token representations again, we use mean-pooling to aggregate all the token representations in a text sample to form sample representation:  h l = Pool ( H l ) , (2) Herein, we get L + 1 layers of sample representations as {  h l } Ll =0 where  h l  R d . With multi-granularity representations, we then need to formulate the specific knowledge we aim to transfer from the teacher to the student. Considering that an element holds its meaning only when it is put into a semantic space where it has 1 https://www.nltk.org/book/ch07.html various relations to other elements, we propose that the knowledge is better specified as the structural relations of the representations in a semantic space, instead of the individual representations themselves. Therefore, instead of directly matching each hidden representation between the teacher and the student, we propose to extract structural relations from multi-granularity representations as the knowledge to teach the student. We first project the representations into multiple sub-spaces, then we extract two types of structural knowledge: pairwise interactions and triplet-wise geometric angles. Multi-head Modeling. A recent study by Wang et al. (2021) pointed out that distilling knowledge with multiple relation heads helps the student learn better. Therefore, before extracting structural knowledge for intermediate representations, we first project them into m sub-spaces, which we call multi-head modeling. Specifically, given a set of n representations R  R n  d , we linearly project them into m sub-spaces whose dimensions are d/m . 2 We use R   R m  n  d/m to denote the multi-head representations which are then used for extracting structural knowledge. Pair-wise Interaction. Given two vectors r i , r j  R d/m in a sub-space, we calculate their interaction as their scaled dot product:  ( r i , r j ) = r i  r  j (cid:112) d/m. Herein, we obtain the multi-head pair-wise interaction features for each pair as P  R m  n  n , where P h,i,j denotes the interaction between the -th representation and the j -th representation in the sub-space of the h -th relation head. Note that P can be considered as the unnormalized self-attention (Vaswani et al., 2017) scores for the given representations, the difference lies in that in our calculation the queries are identical to the keys. Triplet-wise Geometric Angle. Pair-wise interaction features only consider two vectors at once, which is not enough to represent the complicated structural relations between representations in the high-dimensional space. Therefore, we propose to model the high-order relations as the geometric angles for triplets of vectors. Specifically, given 2 For the student model, its representations are linearly projected into intermediate states whose dimensions are the same as the teacher model's hidden dimensions, so that it can be split into m sub-spaces as the teacher model. a triplet of representations r i , r j , r k  , we calculate their geometric angle as:  ( r i , r j , r k ) = cos  r i r j r k =  r ij , r kj  r ij = r i  r j  r i  r j  2 , r kj = r k  r j  r k  r j  2 . (4) We can calculate the geometric angles for all the triplets, and obtain T  R m  n  n  n where T h,i,j,k stands for the angle of  r i r j r k in the sub-space of the h -th relation head. As the computation complexity increases cubically with n , such a calculation is infeasible when the number of representations is large. Hereby, we propose a two-stage selection strategy to sequentially select important representations to form angles. Similar to Goyal et al. (2020), we assume that the more attention a representation receives from others, the more important it is. Therefore, we first calculate the self-attention distributions A  R m  n  n by applying softmax function on the last dimension of P . Then for the j -th representation, we calculate a global salient score s j by summing up self-attention distributions across all heads and all queries. Based on the score, we pick the topk 1 salient representations as vertices. Next, if the i -th representation is selected as vertex, we pick k 2 representations with the highest local salient score to form angles with the vertex. We define the local salient score s i,j as the attention posed by the i -th representation on the j -th representation, The salient scores s i and s i,j are calculated as follows: s j = m (cid:88) h =1 n (cid:88) i =1 A h,i,j , s i,j = m (cid:88) h =1 A h,i,j . Therefore, by sequentially selecting salient representations to form angles, we reduce the computation complexity from O ( mn 3 ) to O ( mk 1 k 2 2 ) . By choosing proper k 2 and k 2 , we can facilitate the computation of triplet-wise geometric angles for any number of representations. We utilize the structural knowledge extraction approach described in Sec. 3.2 to prepare knowledge based on three granularities of representations presented in Sec. 3.1 for distillation. Based on the findings that the bottom layers capture syntactic features while the upper layers encode semantic features (Jawahar et al., 2019), we propose to conduct hierarchical distillation for the student where different granularities of knowledge are transferred to different layers. For a teacher model with L t layers and a student model with L s layers, we first define a layer mapping function g (  ) that maps each student layer to a teacher layer that it learns from. Following previous work (Jiao et al., 2020), we adopt the uniform strategy for g (  ) . Then we transfer token-level and span-level knowledge to the bottomM layers of the student, while leveraging sample-level knowledge to teach its upper L s + 1  M layers. Tokenand Span-level. Specifically, given the token-level and the span-level representations of the teacher { H l t ,  H l t } L t l =0 , we use Eq. 3 and Eq. 4 to calculate the pair-wise interactions and the triplet-wise geometric angles among tokens and spans within a single sample as { P l t ,  P l t } L t l =0 and { T l t ,  T l t } L t l =0 . Similarly, we can obtain the structural relations of the students: { P l s ,  P l s } L s l =0 and { T l s ,  T l s } L s l =0 . We then teach the student by minimizing the differences of the structural relations among their representations between the teacher and the student: L token = (cid:88) 0  l<M (  1 ( P g ( l ) t , P l s ) +  2 ( T g ( l ) t , T l s )) L span = (cid:88) 0  l<M (  1 (  P g ( l ) t ,  P l s ) +  2 (  T g ( l ) t ,  T l s )) . (6) Sample-level. Recall that we obtain {  h lt } L t l =0 and {  h ls } L s l =0 for the teacher and the student where  h lt ,  h ls  R d . Different from the structural knowledge of tokens and spans which is modeled within a sample, the sample-level structural relations rely on a group of sample representations. Although the choice of samples may make a difference to the overall performance, here we simply gather all the sample representations in a mini-batch to calculate their structural relations as the sample-level knowledge. Specifically, we only focus on the triplet-wise relations {  T lt } L t l =0 and {  T ls } L s l =0 : L sample = (cid:88) M  l  L s  2 (  T g ( l ) t ,  T l s ) . 1 and  2 in Eq. 6 and Eq. 7 are loss functions that measure the distance between the structural relations of the teacher's and the student's representations. We empirically choose MSE for  1 and Huber loss (  = 1 ) for  2 . Overall Objectives. The overall distillation objective for multi-granularity structural knowledge distillation is: L 1 =  1 L sample +  2 L token +  3 L span , (8) where  1 ,  2 and  3 are weights of loss functions of different granularities. After this, we also teach the student to match the prediction distributions with the teacher's for text classification tasks: L 2 =  2 DKL ( z t /  z s / ) , (9) where z t and z s are the predicted probability distributions of the teacher and the student respectively,  denotes the temperature. We conduct our experiments on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018). Sepcifically, there are 2 single-sentence tasks: SST-2 (Socher et al., 2013), CoLA (Warstadt et al., 2019), 3 similarity and paraphrase tasks: MRPC (Dolan and Brockett, 2005), STS-B (Cer et al., 2017), QQP (Chen et al., 2018), and 4 inference tasks: MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), RTE (Ben-tivogli et al., 2009), WNLI (Levesque et al., 2012). Following previous work (Jiao et al., 2020; Wang et al., 2021; Park et al., 2021), we evaluate our method on 8 datasets except WNLI. We report accuracy on 5 datasets: SST-2, QQP, MNLI, QNLI and RTE. We report F1 score on MRPC, Matthews correlation coefficient on CoLA, and Spearman's rank correlation coefficient on STS-B. We focus on task-specific distillation. We follow Jiao et al. (2020) to augment the training sets for each of the GLUE tasks using the code 3 they released. We fine-tune ELECTRA base on the original training sets as the teacher model, and utilize TinyBERT-4-312 4 which is distilled on general corpora as the initialization of our student model. For token-level and span-level distillation, we use 64 relation heads for calculating pair-wise interactions, and 1 relation head for triplet-wise angles due to its huge computation and memory costs. And we set k 1 = k 2 = 20 for calculating angles. For sample-level distillation, we use 64 relation heads and set k 1 and k 2 as the batch size. We distill token-level and span-level knowledge to the bottom-2 layers of the student and distill sample-level knowledge to the other layers. For the structural distillation objective, we set  1 = 4 ,  2 =  3 = 1 to maintain their gradient norms in the same order of magnitude. We first distill the student model using Eq. 8 for 50 epochs on CoLA and 20 epochs on other tasks. The learning rate is 1e-5 and the batch size is 32. Then we use Eq. 9 to distill the predictions for all tasks except STS-B since we empirically find that directly fine-tuning after distillation using Eq. 8 yields better performance for it. For QQP and CoLA, we adopt the original training set and distill the student for 10 epochs while for other 5 tasks we use the augmented training sets and distill the student for 3 epochs. We set  as 1.0, the learning 3 https://github.com/huawei-noah/ Pretrained-Language-Model/blob/master/TinyBERT/data_augmentation.py 4 https://huggingface.co/huawei-noah/ TinyBERT_General_4L_312D 1006 Method SST-2 MNLI-(m/mm) MGSKD m=1 92.5 83.6/82.9 MGSKD m=4 92.9 83.9/83.3 MGSKD m=16 93.3 84.3/83.9 MGSKD m=64 93.7 84.7/84.3 MGSKD m=128 93.5 84.8/84.2 Table 2: The impact of relation heads. rate as 1e-5, and the batch size as 32. We release our code to facilitate future research. 5 4.3 Comparison Methods Medium-sized Student Models. Most of the existing knowledge distillation methods are conducted on medium-sized student models which have 6 transformer layers, 768 hidden neurons, 12 attention heads, and overall 66M parameters. We adopt 3 of them as baselines: DistilBERT (Sanh et al., 2019), MiniLMv2 (Wang et al., 2021) and CKD (Park et al., 2021). Notice that these models adopted different distillation settings. DistilBERT and MiniLMv2 were firstly under task-agnostic distillation then directly fine-tuned on GLUE, while CKD was under both task-agnostic and task-specific distillation. The corpora they adopted for task-agnostic distillation were also not exactly the same. Nevertheless, we list the results as they reported on GLUE dev set as baselines, and we implement MiniLMv2 and CKD, two state-of-the-art distillation methods under the same distillation setting as ours for a fair comparison, which is described in the next paragraph. Small-sized Student Models. For fair comparisons, we implement two state-of-the-art distillation methods: MiniLMv2 (Wang et al., 2021), CKD (Park et al., 2021) under the same distillation setting as ours. All these methods use the same student model as ours which has 4 transformer layers, 312 hidden neurons, 12 attention heads and overall 14M parameters. We adopt the fine-tuned ELECTRA base as the teacher, and conduct task-specific distillation using the same distillation schedule and hyperparameters on the same augmented training sets as ours. We first evaluate the effectiveness of our proposed distillation framework. The main results are shown in Table 1. We calculate #Params 5 https://github.com/LC97-pku/MGSKD Method SST-2 MNLI-(m/mm) MGSKD 93.7 84.7/84.3 MGSKD w/o token 93.0 84.1/83.7 MGSKD w/o span 93.2 84.3/84.0 MGSKD w/o sample 92.8 83.9/83.6 MGSKD w token p 92.1 83.4/82.9 MGSKD w token t 91.7 82.8/82.6 MGSKD w token p,t 92.5 83.7/83.2 MGSKD w span p 91.8 82.5/82.3 MGSKD w span t 91.8 82.3/82.0 MGSKD w span p,t 92.2 83.0/82.7 MGSKD w sample p 91.9 82.6/82.5 MGSKD w sample t 92.9 83.9/83.5 MGSKD w sample p,t 92.8 83.7/83.6 Table 3: Ablation study of knowledge granularity. by summing up the number of parameters contained in the embedding layer and all the transformer layers. The speed-up ratios are directly taken from previous works (Jiao et al., 2020; Wang et al., 2021). It can be observed that under the same distillation setting (models with  in Table 1), Student  MGSKD outperforms strong baseline methods (i.e., Student  MiniLMv2 and Student  CKD ) on 7 of the 8 GLUE tasks. When compared with medium-sized models from the literature which have more parameters but under different distillation settings (e.g., CKD), our method can still beat them on the majority of the 8 tasks. And surprisingly, with a stronger teacher model and data augmentation technique, our method MGSKD enables a 14M student transformer model to achieve comparable performance with BERT base on most of the GLUE tasks, while keeping 9.4 times faster. Also, we observe that although MGSKD performs well on most of the GLUE tasks, it lags behind some baselines on CoLA, where the model is asked to judge the grammatical acceptability of a sentence. One reason might be that CoLA requires the model to focus on syntactic information while paying less attention to the sample-level semantic meanings, thus reducing the need for multi-granularity semantic knowledge that we propose to transfer to the student. The Impact of Relation Heads. Recall that when calculating the structural relations between representations, we project them into m relation heads. We show how the number of relation heads impacts the performance on SST-2 and MNLI. As shown in Table 2, the performance gets better as the 1007 8 12 16 20 24 28 32 the choice of k1, k2 (k1=k2) 90.6 90.8 91.0 91.2 91.4 91.6 91.8 SST-2 81.6 81.8 82.0 82.2 82.4 82.6 82.8 MNLI-m different choices of the boundary layer M . number of relation heads increases, since it eases the trouble for the student to learn the structural relations in the very high-dimensional vector space by providing fine-grained supervision in multiple relatively low-dimensional spaces. We also find that when m is large, continuing to increase m is not worthwhile since the time and memory complexity increase linearly with m . Therefore we choose m = 64 in our setting. Ablation Study of Knowledge Granularity. We transfer the structural knowledge to the student in three granularities: token-level, span-level, and sample-level. We extract pair-wise and triplet-wise structural relations for tokenand span-level, while we adopt triplet-wise relations for sample-level. To verify the effectiveness of each granularity of knowledge and each form of structural relations, we conduct ablation studies and present the results in Table 3. (1) We first remove each granularity of knowledge from the objectives of MGSKD individually. 6 We can conclude that the sample-level knowledge is most crucial for the overall performance, the token-level knowledge provides moderate benefit, and the span-level knowledge contributes the least. We assume the reason why span-level knowledge distillation performs a little bit worse than token-level lies in that the average number of meaningful spans per sample on the 8 tasks is 7.19, which is 5.2 times fewer than the average number of tokens. Nevertheless, distillation with span-level knowledge still yields comparable performance. Overall, the results prove that each granularity of knowledge brings a positive effect to the model performance. (2) Then for each granularity, we study the effect of each form of structural knowledge (i.e., pair-wise and triplet-wise 6 When the sample-level objective is removed, we use the remaining objectives for all the student layers instead of only the bottom layers, as this setting yields better performance. relations). In this stage, we distill each granularity of knowledge into all the student layers for a fair comparison. It can be observed that for token-level and span-level knowledge, pair-wise relations are more effective than triplet-wise relations, and the model performs better when jointly utilizing both. While for sample-level knowledge, we find that using triplet-wise relations outperforms using pairwise relations by a large margin. Moreover, jointly utilizing the sample-level pair-wise and triplet-wise relations can't further improve the model's performance, therefore we only employ triplet-wise relations as sample-level knowledge. The Impact of k 1 and k 2 for Calculating Angles. To ease the computation and memory complexity, we propose to sequentially select important representations to form angles, leading to the hyperparameters k 1 and k 2 . We test different choices of k 1 and k 2 by adopting token-level and sample-level triplet-wise relations to teach the student respectively. To reduce the search space, we simply set k 1 = k 2 . We draw the accuracy curve for different choices of k 1 , k 2 , as shown in Fig. 2. For token-level objectives, we find that increasing k 1 , k 2 improves the accuracy when they are small and when k 1 , k 2  20 , the curves begin to vibrate. Therefore we choose k 1 = k 2 = 20 for token-level angle calculation. While for the triplet-wise relations of sample-level features, we observe that the accuracy increases monotonically with k 1 , k 2 . Therefore we just set k 1 , k 2 as the batch size. The Choice of the Boundary Layer M . We propose the hierarchical distillation strategy where we distill the tokenand span-level knowledge into the bottomM layers of the student and transfer the sample-level knowledge to the upper layers. To verify the effectiveness as well as to find the best choice of the boundary layer M , we conduct exper-1008 iments and show the results in Fig. 3. The dashed lines represent the setting dubbed as  all , where we distill token-, spanand sample-level knowledge into all the student layers. And the solid lines denote our hierarchical distillation setting with different choices of the boundary layer M . When M = 0 and M = 4 , the student learns sample-level knowledge or tokenand span-level knowledge for all layers. Without the help of other knowledge granularities, the student yields relatively poor performance on both tasks. As M increases from 0 to 4 , we find the model's performance curves surpass the dashed lines, which verifies the effectiveness of our proposed hierarchical distillation strategy which transfers the knowledge to the proper positions of the student. We find the model achieves the highest accuracy when M = 2 , i.e., the middle layer, indicating that both the syntactic knowledge transferred by tokenand span-level features and the semantic knowledge derived from sample-level features are indispensable. In this paper, we propose a novel knowledge distillation framework named MGSKD. We leverage intermediate representations of multi-granularity language units (i.e., tokens, spans and samples), and form the knowledge as the sophisticated structural relations between the representations rather than the individual representations themselves. The well-organized structural knowledge is then distilled into the student hierarchically across layers. Evaluation results on GLUE benchmark verify the effectiveness of our method. In the future, we plan to explore more forms of structural knowledge. We would like to thank the anonymous reviewers for their constructive comments. This work was supported by the National Key Research and Development Program of China (No. 2020AAA0106600). This paper proposes a knowledge distillation framework that leverages multi-granularity structural knowledge to compress a large and powerful language model into a small one with minimum performance degradation, which is beneficial to energy-efficient NLP applications. The research will not pose ethical problems or negative social consequences. The datasets used in this paper are all publicly available and are widely adopted by researchers as the general testbed for natural language understanding evaluation. The proposed method doesn't introduce ethical/social bias or aggravate the potential bias in the data.
abstain abstain abstain objective objective objective abstain abstain abstain abstain abstain abstain other abstain abstain abstain objective method objective result objective objective method method objective abstain objective objective other other other other other other other abstain other other other other other other other other other other objective method method other method method method method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain method abstain method method abstain method method method abstain abstain method method other method method abstain abstain method method method abstain method method method abstain abstain method abstain method method abstain method method method method method method method method method method method method method method abstain method abstain abstain method method abstain method abstain method method abstain abstain abstain method abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain result result abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain result objective method abstain result objective other other abstain abstain method abstain