We present Knowledge Distillation with Meta Learning (MetaDistil), a simple yet effective alternative to traditional knowledge distillation (KD) methods where the teacher model is fixed during training. We show the teacher network can learn to better transfer knowledge to the student network (i.e., learning to teach ) with the feedback from the performance of the distilled student network in a meta learning framework. Moreover, we introduce a pilot update mechanism to improve the alignment between the inner-learner and meta-learner in meta learning algorithms that focus on an improved inner-learner. Experiments on various benchmarks show that MetaDistil can yield significant improvements compared with traditional KD algorithms and is less sensitive to the choice of different student capacity and hy-perparameters, facilitating the use of KD on different tasks and models. 1 1 Introduction With the prevalence of large neural networks with millions or billions of parameters, model compression is gaining prominence for facilitating efficient, eco-friendly deployment for machine learning applications. Among techniques for compression, knowledge distillation (KD) (Hinton et al., 2015b) has shown effectiveness in both Computer Vision and Natural Language Processing tasks (Hinton et al., 2015b; Romero et al., 2015; Zagoruyko & Komodakis, 2017; Tung & Mori, 2019; Peng et al., 2019; Ahn et al., 2019; Park et al., 2019; Passalis & Tefas, 2018; Heo et al., 2019; Kim et al., 2018; Shi et al., 2021; Sanh et al., 2019; Jiao et al., 2019; Wang et al., 2020b). Previous works often train a large model as the teacher; then they fix the teacher and train a student model to mimic the  Equal contribution. However, this paradigm has the following drawbacks: (1) The teacher is unaware of the student's capacity. Recent studies in pedagogy suggest student-centered learning, which considers students' characteristics and learning capability, has shown effectiveness improving students' performance (Cornelius-White, 2007; Wright, 2011). However, in conventional knowledge distillation, the student passively accepts knowledge from the teacher, without regard for the student model's learning capability and performance. Recent works (Park et al., 2021; Shi et al., 2021) introduce student-aware distillation by jointly training the teacher and the student with task-specific objectives. However, there is still space for improvement since: (2) The teacher is not optimized for distillation. In previous works, the teacher is often trained to optimize its own inference performance. However, the teacher is not aware of the need to transfer its knowledge to a student and thus usually does so suboptimally. A real-world analogy is that a PhD student may have enough knowledge to solve problems themselves, but requires additional teaching training to qualify as a professor. To address these two drawbacks, we propose Knowledge Distillation with Meta Learning (MetaDistil), a new teacher-student distillation framework using meta learning (Finn et al., 2017) to exploit feedback about the student's learning progress to improve the teacher's knowledge transfer ability throughout the distillation process. On the basis of previous formulations of bi-level optimization based meta learning (Finn et al., 2017), we propose a new mechanism called pilot update that aligns the learning of the bi-level learners (i.e., the teacher and the student). We illustrate the workflow of MetaDistil in Figure 1. The teacher in MetaDistil is trainable, which enables the teacher to adjust to its student network and also improves its 7037 (1) Teaching experiment (2) Quiz & Meta update (3) Knowledge distillation T S' Training Batches S Copy Update T S' Training Batches Quiz Samples LCE <latexit sha1_base64="cWzYMh3AeRJkO+vsctAQMH7i188=">AAACAnicbVBNS8NAEN3Ur1q/op7ES7AInkpSBT0Wi+DBQwXbCm0Im+22XbrZhN2JUELw4l/x4kERr/4Kb/4bN2kO2vpg4e17M8zM8yPOFNj2t1FaWl5ZXSuvVzY2t7Z3zN29jgpjSWibhDyU9z5WlDNB28CA0/tIUhz4nHb9STPzuw9UKhaKO5hG1A3wSLAhIxi05JkH/QDDmGCe3KRekn8YJM2rNPXMql2zc1iLxClIFRVoeeZXfxCSOKACCMdK9Rw7AjfBEhjhNK30Y0UjTCZ4RHuaChxQ5Sb5Cal1rJWBNQylfgKsXP3dkeBAqWng68psRzXvZeJ/Xi+G4YWbMBHFQAWZDRrG3ILQyvKwBkxSAnyqCSaS6V0tMsYSE9CpVXQIzvzJi6RTrzmntfrtWbVxWcRRRofoCJ0gB52jBrpGLdRGBD2iZ/SK3own48V4Nz5mpSWj6NlHf2B8/gAT8Jfd</latexit> T S Training Batches Update Update Forward pass Backward pass: 1st derivatives 2nd derivatives Pilot update L 0 KD <latexit sha1_base64="uziwHVw4qqKfwsWyy8k1x7MuDt4=">AAACA3icbVDLSsNAFL3xWesr6k43wSK6KkkVdFnQhaCLCvYBbQiT6bQdOpmEmYlQQsCNv+LGhSJu/Ql3/o2TNAttPTBw5px7ufceP2JUKtv+NhYWl5ZXVktr5fWNza1tc2e3JcNYYNLEIQtFx0eSMMpJU1HFSCcSBAU+I21/fJn57QciJA35vZpExA3QkNMBxUhpyTP3ewFSI4xYcnucekn+oyq5uUpTz6zYVTuHNU+cglSgQMMzv3r9EMcB4QozJGXXsSPlJkgoihlJy71YkgjhMRqSrqYcBUS6SX5Dah1ppW8NQqEfV1au/u5IUCDlJPB1ZbajnPUy8T+vG6vBhZtQHsWKcDwdNIiZpUIrC8TqU0GwYhNNEBZU72rhERIIKx1bWYfgzJ48T1q1qnNard2dVeq1Io4SHMAhnIAD51CHa2hAEzA8wjO8wpvxZLwY78bHtHTBKHr24A+Mzx+AZ5gF</latexit> L 0 KD <latexit sha1_base64="uziwHVw4qqKfwsWyy8k1x7MuDt4=">AAACA3icbVDLSsNAFL3xWesr6k43wSK6KkkVdFnQhaCLCvYBbQiT6bQdOpmEmYlQQsCNv+LGhSJu/Ql3/o2TNAttPTBw5px7ufceP2JUKtv+NhYWl5ZXVktr5fWNza1tc2e3JcNYYNLEIQtFx0eSMMpJU1HFSCcSBAU+I21/fJn57QciJA35vZpExA3QkNMBxUhpyTP3ewFSI4xYcnucekn+oyq5uUpTz6zYVTuHNU+cglSgQMMzv3r9EMcB4QozJGXXsSPlJkgoihlJy71YkgjhMRqSrqYcBUS6SX5Dah1ppW8NQqEfV1au/u5IUCDlJPB1ZbajnPUy8T+vG6vBhZtQHsWKcDwdNIiZpUIrC8TqU0GwYhNNEBZU72rhERIIKx1bWYfgzJ48T1q1qnNard2dVeq1Io4SHMAhnIAD51CHa2hAEzA8wjO8wpvxZLwY78bHtHTBKHr24A+Mzx+AZ5gF</latexit> LKD <latexit sha1_base64="Ok//7Q4UXi0+yvIX1pX7vfyqMZs=">AAACAnicbVDLSsNAFJ3UV62vqCtxEyyCq5JUQZcFXQi6qGAf0IYwmU7aoZNJmLkRSghu/BU3LhRx61e482+ctFlo64GBM+fcy733+DFnCmz72ygtLa+srpXXKxubW9s75u5eW0WJJLRFIh7Jro8V5UzQFjDgtBtLikOf044/vsz9zgOVikXiHiYxdUM8FCxgBIOWPPOgH2IYEczT28xLpx8G6c1Vlnlm1a7ZU1iLxClIFRVoeuZXfxCRJKQCCMdK9Rw7BjfFEhjhNKv0E0VjTMZ4SHuaChxS5abTEzLrWCsDK4ikfgKsqfq7I8WhUpPQ15X5jmrey8X/vF4CwYWbMhEnQAWZDQoSbkFk5XlYAyYpAT7RBBPJ9K4WGWGJCejUKjoEZ/7kRdKu15zTWv3urNqoF3GU0SE6QifIQeeoga5RE7UQQY/oGb2iN+PJeDHejY9ZackoevbRHxifPxnSl9Q=</latexit> Figure 1: The workflow of MetaDistil. teaching skills. Motivated by the idea of student-centered learning, we allow the teacher to adjust its output based on the performance of the student model on a quiz set, which is a separate reserved data split from the original training set. For each training step, we first copy the student S to S (cid:48) and update S (cid:48) by a common knowledge distillation loss. We call this process a teaching experiment. In this way, we can obtain an experimental student S (cid:48) that can be quizzed. Then, we sample from the quiz set, and calculate the loss of S (cid:48) on these samples. We use this loss as a feedback signal to meta-update the teacher by calculating second derivatives and performing gradient descent (Finn et al., 2017). Finally, we discard the experimental subject S (cid:48) and use the updated teacher to distill into the student S on the same training batches. The use of meta learning allows the teacher model to receive feedback from the student in a completely differentiable way. We provide a simple and intuitive approach to explicitly optimize the teacher using the student's quiz performance as a proxy. To test the effectiveness of MetaDistil, we conduct extensive experiments on text and image classification tasks. MetaDistil outperforms knowledge distillation by a large margin, verifying the effectiveness and versatility of our method. Also, our method achieves state-of-the-art performance compressing BERT (Devlin et al., 2019) on the GLUE benchmark (Wang et al., 2019) and shows competitive results compressing ResNet (He et al., 2016) and VGG (Simonyan & Zisserman, 2015) on CIFAR-100 (Krizhevsky et al., 2009). Additionally, we design experiments to analyze and explain the improvement. Ablation studies show the effectiveness of our proposed pilot update and dynamic distillation. Also, compared to conventional KD, MetaDistil is more robust to different student capacity and hyperparameters, which is probably because of its ability to adjust the parameters of the teacher model. Knowledge Distillation Recently, many attempts have been made to accelerate large neural networks (Xu et al., 2020, 2021b; Zhou et al., 2020, 2021; Xu & McAuley, 2022). Knowledge distillation is a prominent method for training compact networks to achieve comparable performance to a deep network. Hinton et al. (2015b) first introduced the idea of knowledge distillation to exploit the dark knowledge (i.e., soft label distribution) from a large teacher model as additional supervision for training a smaller student model. Since its introduction, several works (Romero et al., 2015; Zagoruyko & Komodakis, 2017; Tung & Mori, 2019; Park et al., 2019; Sun et al., 2019; Jiao et al., 2019) have investigated methods that align different latent representations between the student and teacher models for better knowledge transfer. In the context of knowledge distillation, MetaDistil shares some common ideas with the line of work that utilizes a sequence of intermediate teacher models to make the teacher network better adapt to the capacity of the student model throughout the training process, including teacher assistant knowledge distillation (TAKD) (Mirzadeh et al., 2020) and route constraint optimization (RCO) (Jin et al., 2019). However, the intermediate teachers are heuristically selected independently of the training process and the evolution of the teacher network is discrete. In contrast, MetaDistil employs meta learning to make the teacher model adapt to the current state of the student model and provide a continuously 7038 evolving meta-teacher that can better teach the student. Concurrently, Park et al. (2021) and Shi et al. (2021) propose to update the teacher model jointly with the student model with task specific objectives (e.g., cross-entropy loss) during the KD process and add constraints to keep student and teacher similar to each other. Their approaches makes the teacher model aware of the student model by constraining the teacher model's capacity. However, the teacher models in their methods are still not optimized for knowledge transfer. In addition, Zhang et al. (2018) introduced deep mutual learning where multiple models learn collaboratively and teach each other throughout the training process. While it is focused on a different setting where different models have approximately the same capacity and are learned from scratch, it also encourages the teacher model to behave similarly to the student model. Different from all aforementioned methods, MetaDistil employs meta learning to explicitly optimize the teacher model for better knowledge transfer ability, and leads to improved performance of the resulting student model. Meta Learning The core idea of meta learning is learning to learn, which means taking the optimization process of a learning algorithm into consideration when optimizing the learning algorithm itself. Meta learning typically involves a bi-level optimization process where the inner-learner provides feedback for optimization of the meta-learner. Successful applications of meta learning include learning better initialization (Finn et al., 2017), architecture search (Liu et al., 2019), learning to optimize the learning rate schedule (Baydin et al., 2018), and learning to optimize (Andrychowicz et al., 2016). These works typically aim to obtain an optimized meta-learner (i.e., the teacher model in MetaDistil), while the optimization of the inner-learner (i.e., the student model in MetaDis-til), is mainly used to provide learning signal for the meta optimization process. This is different from the objective of knowledge distillation where an optimized student model is the goal. Recently, there have been a few works investigating using this bi-level optimization framework to obtain a better inner-learner. For example, meta pseudo labels (Pham et al., 2020) use meta learning to optimize a pseudo label generator for better semi-supervised learning; meta back-translation (Pham et al., 2021) meta-trains a back-translation model to better train a machine translation model. These methods adapt the same bi-level optimization process as previous works where the goal is to obtain an optimized meta-learner. In these approaches, during each iteration, the meta-learner is optimized for the original inner-learner and then applied to the updated inner-learner in the next iteration. This leads to a mismatch between the meta-learner and the inner-learner, and is therefore suboptimal for learning a good inner-learner. In this paper, we introduce a pilot update mechanism, which is a simple and general method for this kind of problems, for the inner-learner to mitigate this issue and make the updated meta-learner better adapted to the inner-learner. Meta Knowledge Distillation Recently, some works on KD take a meta approach. Pan et al. (2020) proposed a framework to train a meta-teacher across domains that can better fit new domains with meta-learning. Then, traditional KD is performed to transfer the knowledge from the meta-teacher to the student. Liu et al. (2020) proposed a self-distillation network which utilizes meta-learning to train a label-generator as a fusion of deep layers in the network, to generate more compatible soft targets for shallow layers. Different from the above, MetaDistil is a general knowledge distillation method that exploits meta-learning to allow the teacher to learn to teach dynamically. Instead of merely training a meta-teacher, our method uses meta-learning throughout the procedure of knowledge transfer, making the teacher model compatible for the student model for every training example during each training stage. An overview of MetaDistil is presented in Figure 1. MetaDistil includes two major components. First, the meta update enables the teacher model to receive the student model's feedback on the distillation process, allowing the teacher model to learn to teach and provide distillation signals that are more suitable for the student model's current capacity. The pilot update mechanism ensures a finer-grained match between the student model and the meta-updated teacher model. Knowledge distillation algorithms aim to exploit the hidden knowledge from a large teacher network, denoted as T , to guide the training of a shallow student network, denoted as S . To help transfer the knowledge from the teacher to the student, apart from the original task-specific objective (e.g., cross-entropy loss), a knowledge distillation objective which aligns the behavior of the student and the teacher is included to train the student network. Formally, given a labeled dataset D of N samples D = { ( x 1 , y 1 ) , . . . , ( x N , y N ) } , we can write the loss function of the student network as follows, LS ( D ;  S ;  T ) = 1 NN (cid:88) i =1 [  LT ( y i , S ( x i ;  S )) + (1   ) LKD ( T ( x i ;  T ) , S ( x i ;  S ))] (1) where  is a hyper-parameter to control the relative importance of the two terms;  T and  S are the parameters of the teacher T and student S , respectively. LT refers to the task-specific loss and LKD refers to the knowledge distillation loss which measures the similarity of the student and the teacher. Some popular similarity measurements include the KL divergence between the output probability distribution, the mean squared error (MSE) between student and teacher logits, the similarity between the student and the teacher's attention distribution, etc. We do not specify the detailed form of the loss function because MetaDistil is a general framework that can be easily applied to various kinds of KD objectives as long as the objective is differentiable with respect to the teacher parameters. In the experiments of this paper, we use mean squared error between the hidden states of the teacher and the student for both our method and the KD baseline since recent study Kim et al. (2021) finds that it is more stable and slightly outperforms than KL divergence. In meta learning algorithms that involve a bi-level optimization problem (Finn et al., 2017), there exists an inner-learner f i and a meta-learner f m . The inner-learner is trained to accomplish a task T or a distribution of tasks with help from the meta-learner. The training process of f i on T with the help of f m is typically called inner-loop , and we can denote f (cid:48) i ( f m ) as the updated inner-learner after the inner-loop. We can express f (cid:48) i as a function of f m because learning f i depends on f m . In return, the meta-learner is optimized with a meta objective, which is generally the maximization of expected performance of the inner-learner after the inner-loop, i.e., f (cid:48) i ( f m ) . This learning process is called a meta-loop and is often accomplished by gradient descent with derivatives of L ( f (cid:48) i ( f m )) , the loss of updated inner-leaner on some held-out support set (i.e., the quiz set in our paper). 3.2.1 Pilot Update In the original formulation of meta learning (Finn et al., 2017), the purpose is to learn a good meta-learner f m that can generalize to different inner-learners f i for different tasks. In their approach, the meta-learner is optimized for the original inner-learner at the beginning of each iteration and the current batch of training data. The updated meta-learner is then applied to the updated inner-learner and a different batch of data in the next iteration. This behavior is reasonable if the purpose is to optimize the meta-learner. However, in MetaDistil, we only care about the performance of the only inner-learner, i.e., the student. In this case, this behavior leads to a mismatch between the meta-learner and the inner-learner, and is therefore suboptimal for learning a good inner-learner. Therefore, we need a way to align and synchronize the learning of the metaand inner-learner, in order to allow an update step of the meta-learner to have an instant effect on the inner-learner. This instant reflection prevents the meta-learner from catastrophic forgetting (McCloskey & Cohen, 1989). To achieve this, we design a pilot update mechanism. For a batch of training data x , we first make a temporary copy of the inner-learner f i and update both the copy f (cid:48) i and the meta learner f m on x . Then, we discard f (cid:48) i and update f i again with the updated f m on the same data x . This mechanism can apply the im-pact of data x to both f m and f i at the same time, thus aligns the training process. Pilot update is a general technique that can potentially be applied to any meta learning application that optimizes the inner-learner performance. We will describe how we apply this mechanism to MetaDistil shortly and empirically verify the effectiveness of pilot update in Section 4.2. In MetaDistil, we would like to optimize the teacher model, which is fixed in traditional KD frameworks. Different from previous deep mutual learning (Zhang et al., 2018) methods that switch the role between the student and teacher 7040 Algorithm 1 Knowledge Distillation with Meta Learning (MetaDistil) Require: student  S , teacher  T , train set D , quiz set Q Require:  ,  : learning rate for the student and the teacher 1: while not done do 2: Sample batch of training data x  D 3: Copy student parameter  S to student  (cid:48) S 4: Update  (cid:48) S with x and  T :  (cid:48) S   (cid:48) S     (cid:48) SLS ( x ;  S ;  T ) 5: Sample a batch of quiz data q  Q 6: Update  T with q and  (cid:48) S :  T   T     TLT ( q ,  (cid:48) S (  T )) 7: Update original  S with x and the updated  T :  S   S     SLS ( x ;  S ;  T ) 8: end while network and train the original teacher model with soft labels generated by the student model, or recent works (Shi et al., 2021; Park et al., 2021) that update the teacher model with a task-specific loss during the KD process, MetaDistil explicitly optimizes the teacher model in a learning to teach fashion, so that it can better transfer its knowledge to the student model. Concretely, the optimization objective of the teacher model in the MetaDistil framework is the performance of the student model after distilling from the teacher model. This learning to teach paradigm naturally fits the bi-level optimization framework in meta learning literature. In the MetaDistil framework, the student network  S is the inner-learner and the teacher network  T is the meta-learner. For each training step, we first copy the student model  S to an experi-mental student  (cid:48) S . Then given a batch of training examples x and the learning rate  , the experimental student is updated in the same way as conventional KD algorithms:  (cid:48) S (  T ) =  S     SLS ( x ;  S ;  T ) . To simplify notation, we will consider one gradient update for the rest of this section, but using multiple gradient updates is a straightforward extension. We observe that the updated experimental student parameter  (cid:48) S , as well as the student quiz loss l q = LT ( q ,  (cid:48) S (  T )) on a batch of quiz samples q sampled from a held-out quiz set Q , is a function of the teacher parameter  T . Therefore, we can optimize l q with respect to  T by a learning rate  :  T   T     TLT (cid:0) q ,  (cid:48) S (  T ) (cid:1) (3) We evaluate the performance of the experimental student on a separate quiz set to prevent overfitting the validation set, which is preserved for model selection. Note that the student is never trained on the quiz set and the teacher only performs meta-update on the quiz set instead of fitting it. We do not use a dynamic quiz set strategy because otherwise the student would have been trained on the quiz set and the loss would not be informative. After meta-updating the teacher model, we then update the real student model in the same way as described in Equation 2. Intuitively, optimizing the teacher network  T with Equation 3 is maximizing the expected performance of the student network after being taught by the teacher with the KD objective in the inner-loop. This meta-objective allows the teacher model to adjust its parameters to better transfer its knowledge to the student model. We apply the pilot update strategy described in Section 3.2.1 to better align the learning of the teacher and student, as shown in Algorithm 1. 4 Experiments 4.1 Experimental Setup We evaluate MetaDistil on two commonly used classification benchmarks for knowledge distillation in both Natural Language Processing and Computer Vision (see Appendix A). Settings For NLP, we evaluate our proposed approach on the GLUE benchmark (Wang et al., 2019). Specifically, we test on MRPC (Dolan & Brockett, 2005), QQP and STS-B (Conneau & Kiela, 2018) for Paraphrase Similarity Matching; SST-2 (Socher et al., 2013) for Sentiment Classification; MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016) and RTE (Wang et al., 2019) for the Natural Language Inference; CoLA (Warstadt et al., 2019) for Linguistic Acceptability. Following previous studies (Sun et al., 2019; Jiao et al., 2019; Xu et al., 2020), our goal is to distill BERT-Base (Devlin et al., 2019) into a 6-layer BERT with the hidden size of 768. We use MSE loss between model logits as the distillation objective. The reported results are in the same format as on the GLUE leaderboard. For MNLI, 7041 Method #Param. we report the results on MNLI-m and MNLI-mm, respectively. For MRPC and QQP, we report both F1 and accuracy. For STS-B, we report Pearson and Spearman correlation. The metric for CoLA is Matthew's correlation. The other tasks use accuracy as the metric. Following previous works (Sun et al., 2019; Turc et al., 2019; Xu et al., 2020), we evaluate MetaDistil in a task-specific setting where the teacher model is fine-tuned on a downstream task and the student model is trained on the task with the KD loss. We do not choose the pretraining distillation setting since it requires significant computational resources. We implement MetaDistil based on Hugging Face Transformers (Wolf et al., 2020). Baselines For comparison, we report the results of vanilla KD and patient knowledge distillation (Sun et al., 2019). We also include the results of progressive module replacing (Xu et al., 2020), a state-of-the-art task-specific compression method for BERT which also uses a larger teacher model to improve smaller ones like knowledge distillation. In addition, according to Turc et al. (2019), the reported performance of current task-specific BERT compression methods is underestimated because the student model is not appropriately initialized. To ensure fair comparison, we re-run task-specific baselines with student models 7042 initialized by a pretrained 6-layer BERT model and report our results in addition to the official numbers in the original papers. We also compare against deep mutual learning (DML) (Zhang et al., 2018), teacher assistant knowledge distillation (TAKD) (Mirzadeh et al., 2020), route constraint optimization (RCO) (Jin et al., 2019), proximal knowledge teaching (ProKT) (Shi et al., 2021), and student-friendly teacher network (SFTN) (Park et al., 2021), where the teacher network is not fixed. For reference, we also present results of pretraining distilled models including DistilBERT (Sanh et al., 2019), TinyBERT (Jiao et al., 2019), MiniLM v1 and v2 (Wang et al., 2020b,a). Note that among these baselines, PKD (Sun et al., 2019) and Theseus (Xu et al., 2020) exploit intermediate features while TinyBERT and the MiniLM family use both intermediate and Transformer-specific features. In contrast, MetaDistil uses none of these but the vanilla KD loss (Equation 1). Training Details For training hyperparameters, we fix the maximum sequence length to 128 and the temperature to 2 for all tasks. For our method and all baselines (except those with officially reported numbers), we perform grid search over the sets of the student learning rate  from {1e-5, 2e-5, 3e-5}, the teacher learning rate  from {2e-6, 5e-6, 1e-5}, the batch size from {32, 64}, the weight of KD loss from {0.4, 0.5, 0.6}. We randomly split the original training set to a new training set and the quiz set by 9 : 1 . For RCO, we select four unconverged teacher checkpoints as the intermediate training targets. For TAKD, we use KD to train a teacher assistant model with 10 Transformer layers. We report the experimental results on both the development set and test set of the eight GLUE tasks (Wang et al., 2019) in Table 1. MetaDistil achieves state-of-the-art performance under the task-specific setting and outperforms all KD baselines. Notably, without using any intermediate or model-specific features in the loss function, MetaDistil outperforms methods with carefully designed features, e.g., PKD and TinyBERT (without data augmentation). Compared with other methods with a trainable teacher (Zhang et al., 2018; Mirzadeh et al., 2020; Jin et al., 2019; Shi et al., 2021), our method still demonstrates superior performance. As we analyze, with the help of meta learning, MetaDistil is able to directly optimize the teacher's teaching ability thus yielding a further improvement in terms of student accuracy. Also, we observe a performance drop by replacing pilot update with a normal update. This ablation study verifies the effectiveness of our proposed pilot update mechanism. Moreover, MetaDistil achieves very competitive results on image classification as well, as described in Section A.2. We investigate the effect of meta-update for each iteration. We inspect (1) the validation loss of S (cid:48) after the teaching experiment and that of S after the real distillation update, and (2) the KD loss, which describes the discrepancy between student and teacher, before and after the teacher update. We find that for 87% of updates, the student model's validation loss after real update (Line 7 in Algorithm 1) is smaller than that after the teaching experiment (Line 4 in Algorithm 1), which would be the update to the student S in the variant without pilot update. This confirms the effectiveness of the pilot update mechanism on better matching the student and teacher model. Moreover, we find that in 91% of the first half of the updates, the teacher becomes more similar (in terms of logits distributions) to the student after the meta-update, which indicates that the teacher is learning to adapt to a low-performance student (like an elementary school teacher). However, in the second half of MetaDistil, this percentage drops to 63%. We suspect this is because in the later training stages, the teacher needs to actively evolve itself beyond the student to guide the student towards further improvement (like a university professor). Finally, we try to apply a meta-learned teacher to a conventional static distillation and also to an unfamiliar student. We describe the results in details in Section A.3. A motivation of MetaDistil is to enable the teacher to dynamically adjust its knowledge transfer in an optimal way. Similar to Adam (Kingma & Ba, 2015) vs. SGD (Sinha & Griscik, 1971; Kiefer et al., 1952) for optimization, with the ability of dynamic adjusting, it is natural to expect MetaDistil to be more insensitive and robust to changes of the settings. Here, we evaluate the performance of MetaDistil with students of various capability, and 7043 75 80 A v g . Student Capability To investigate the performance of MetaDistil under different student capacity, we experiment to distill BERT-Base into BERT-6L, Medium, Small, Mini and Tiny (Turc et al., 2019) with conventional KD and MetaDistil. We plot the performance with the student's parameter number in Figure 2. Additionally, we show results for different compression ratio in Appendix B. Loss Weight In KD, tuning the loss weight is nontrivial and often requires hyperparameter search. To test the robustness of MetaDistil under different loss weights, we run experiments with different  (Equation 1). As shown in Figure 3, MetaDistil consistently outperforms conventional KD and is less sensitive to different  . Temperature Temperature is a re-scaling trick introduced in Hinton et al. (2015b). We try different temperatures and illustrate the performance of KD and MetaDistil in Figure 4. MetaDistil shows better performance and robustness compared to KD. Like all meta learning algorithms, MetaDistil inevitably requires two rounds of updates involving both first and second order derivatives. Thus, MetaDistil requires additional computational time and memory than a normal KD method, which can be a limitation of our method. We compare the Method PKD (2019) ProKT (2021) MetaDistil (ours) Training Time (Best) 13 min. 25 min. 31 min. computational overheads of MetaDistil with other methods in Table 2. Although our approach takes more time to achieve its own peak performance, it can match up the performance of PKD (Sun et al., 2019) with a similar time cost. The memory use of our method is higher than PKD and ProKT (Shi et al., 2021). However, this one-off investment can lead to a better student model for inference, thus can be worthy. In this paper, we present MetaDistil, a knowledge distillation algorithm powered by meta learning that explicitly optimizes the teacher network to better transfer its knowledge to the student network. The extensive experiments verify the effectiveness and robustness of MetaDistil. MetaDistil focuses on improving the performance of knowledge distillation and does not introduce extra ethical concerns compared to vanilla KD methods. Nevertheless, we would like to point out that as suggested by Hooker et al. (2020), model compression may lead to biases. However, this is not an outstanding problem of our method but a common risk in model compression, which needs to be addressed in the future. We would like to thank the anonymous reviewers and the area chair for their insightful comments. This project is partly supported by NSF Award #1750063.
method result result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain objective objective abstain result method method method abstain method method result result method objective abstain other other other other other other other other other other other other other other other other other other other other other other other method other other other other other method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method other abstain abstain abstain abstain abstain method abstain method abstain method method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain method result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain other other