Although pre-trained big models ( e.g. , BERT, ERNIE, XLNet, GPT3 etc.) have delivered top performance in Seq2seq modeling, their deployments in real-world applications are often hindered by the excessive computations and memory demand involved. For many applications, including named entity recognition (NER), matching the state-of-the-art result under budget has attracted considerable attention. Drawing power from the recent advance in knowledge distillation (KD), this work presents a novel distillation scheme to efficiently transfer the knowledge learned from big models to their more affordable counterpart. Our solution highlights the construction of surrogate labels through the k -best Viterbi algorithm to distill knowledge from the teacher model. To maximally assimilate knowledge into the student model, we propose a multi-grained distillation scheme, which integrates cross entropy involved in conditional random field (CRF) and fuzzy learning. To validate the effectiveness of our proposal, we conducted a comprehensive evaluation on five NER benchmarks, reporting cross-the-board performance gains relative to competing prior-arts. We further discuss ablation results to dissect our gains. The task of named entity recognition (NER) aims to put named entity mentioned in a sentence into some pre-defined categories such as the person names, organizations, locations, etc. NER is a fundamental task in nature language processing (NLP), and often serves as an information extraction tool embedded in solutions for downstream tasks such as event recognition and aspect-level sentiment analysis. And therefore, the computational and memory efficiency of an NER model is often considered crucial in many empirical settings. years. One fruitful direction is to boost NER performance through the use of more sophisticated model architectures, such as Transformer and its variants ( e.g. , BERT (Devlin et al., 2019), ERNIE (Sun et al., 2019c), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019b), GPT3 (Brown et al., 2020), etc.). Many of these models are based on the attention mechanism, which allows the models to adaptively focus on different parts of the sentence based on its current understanding. This enables more accurate parsing of the context, which is critical for the NER task. While such advanced models have delivered substantial improvements, a major drawback is that they are typically computationally expensive and memory intensive, preventing their applications in many cost-sensitive settings. As such, it is often desirable to reduce those big models into more affordable counterparts, preferably without any significant performance drop. One strategy is to compress or truncate the original model, examples in this category include parameter pruning (Srinivas and Babu, 2015; McCarley, 2019), low-rank approximation (Yu et al., 2017; Ma et al., 2019) and parameter quantization (Gong et al., 2014; Wu et al., 2016). The resulting model has a similar architecture to the original model, and consequently may suffer from similar limitations of the original model. Alternatively, knowledge distillation (KD) considers transferring knowledge into models with heterogeneous architectures (Hin-ton et al., 2015), which allows more flexibility in the control of resource usage. In KD, a new learner, often dubbed the student model, assimilates knowledge from a pre-trained model, commonly known as the teacher. For classification tasks, this is typically achieved via minimizing their discrepancy in the output, regardless of the internal model architectures. Teacher outputs, sometimes referred to as the soft-labels, typically encode more information than what a student might receive from the raw training feature-label pairs, thus resulting the improved learning efficiency. However, for sequence tasks such as NER, standard techniques from KD do not readily apply. While common KD seeks to minimize the KL divergence between the output label distributions, for NER, the number of label combinations grows exponentially wrt the sequence length. Extracting teacher knowledge as if each combination is a different label category would be largely inefficient, if possible at all. On the other hand, many SOTA NER models are built on conditional random fields (CRF) to incorporate the label dependencies, and it only offers an un-normalized likelihood that is not directly amenable to the computation of KL-divergence. Another challenge for training an NER model is the lack of labeled sequences for training, while the unlabeled sequences may be plentiful. The promise to leverage unlabeled data to improve NER accuracy is appealing. To address the above challenges, in this paper, we present an efficient knowledge distillation scheme that trains a light model ( e.g. , BiLSTM) which is able to retain the accuracy of its heavier counterparts, such as BERT, while significantly reduce cost. Our solution exploits the soft sur-rogates, i.e. , the most probable label sequences under the teacher model, to inform the student learner. To efficiently identify the most likely label sequences and determine their relative likelihood, we explore the use of Viterbi algorithm to expedite computation. We also explore the use of unlabeled data to improve performance. In summary, we highlight the following contributions in this study: ( i ) We present a novel multi-grained knowledge distillation strategy for sequence labeling via efficiently selecting k -best label sequence using Viterbi algorithm; ( ii ) We advocate the use of a complete cross entropy loss and fuzzy distillation loss to respectively account for probability mass of un-selected sequences and the uncertainties in teacher confidence; ( iii ) We present a comprehensive empirical analysis to dissect the gains from each individual components. We also show our model delivers substantial improvement relative to competing solutions on a wide-range of real-world benchmarks to demonstrate its utility. Our work is inspired by three lines of research: sequence-level knowledge distillation, k best best Viterbi algorithm and fuzzy conditional random field (CRF). In the following we review the technical backgrounds that are needed for the construction of our model. Sequence-level Knowledge Distillation. Knowledge distillation (KD) is originally developed for classification tasks (Hinton et al., 2015; Tang et al., 2019). When dealing with sequence outputs ( e.g. , machine translation, sequence labeling), where each unique combination of the output sequence is treated as different category, then the standard distillation objective is no longer appropriate, if feasible at all. This is because the number of unique combinations for a length L sequence with T possible tags scale at TL . To combat such exponential scaling, Kim and Rush (2016) investigated using beam search for teacher output to select k -best candidates for KD in neural machine translation (NMT), and Mun'im et al. (2019) utilized an similar technique for KD in Large Vocabulary Continuous Speech Recognition (LVSCR) tasks. k -best Viterbi Decoding. Viterbi decoding is a dynamic programming technique to find a sequence with the highest score in an exponential-growth do-main, with only linear complexity (Viterbi, 1967). Generalization has been proposed to extend its original scope to find the topK sequences that are most probable, see (Huang and Chiang, 2005; Nielsen, 2011) for details. A summary of the algorithm can be found in the supplementary material. In the context of KD in sequence tasks, a trained teacher model assigns varying probability to all sequence combinations. Our motivation is that the k -best Viterbi can be repurposed to pick out the K -most probable label sequences predicted by the teacher model, which plays an analogous role to the soft-labels (Figure 1a), without incurring unmanageable computational overhead. Conditional Random Field. CRF (Lafferty et al., 2001) is a classic and powerful energy-based model that is capable of capturing complex spatial or temporal dependency structures. In the context of NLP, it has been generally used as a refinement layer that accounts for correlations missed by the base NLP model, which typically brings in additional performance gains for various NLP tasks (Lafferty et al., 2001; Collobert et al., 2011a; Huang et al., 2015). Implementation-wise, CRF computes an energy given a candidate output y and a context x ( i.e. , input sequence), followed by a softmax operator to obtain the conditional likelihood, i.e. , 1 5 10 20 50 K 85 90 95 F 1 v a l u e MSRACoNLL-2003OntoNotes 4.0 1 5 10 15 20 25 30 35 40 45 50 K 0 5 10 15 20 25 P r opo r ti on ( % ) Figure 1: Left. The model prediction accuracy improves as one includes more candidate K . Right. The non-topK probability remains non-neglectable even for moderately large K . Here, OntoNotes 4.0 is used for illustration, and Y-axis represents the proportion of samples, whose non-topK cumulative probability is greater than 0 . 1 . where Y all is the set of all possible tag sequences, s ( x , y ) represents the compliance energy score between two sequences x and y . In classical NLP models, s ( x , y ) is typically specified via handcrafted features and dependencies. In modern deep learning models, a typical decomposition of s ( x , y ) is given by two parts s ( x , y ) = L  1 (cid:88) j =1 logit j, y j ( x ) + A y j , y j +1 , (2) namely the emission and transition score. The transition matrix A  RT  T characterizes the smoothness of the label sequence (probability of switching between consequent labels), and logit j,k ( x ) denotes the likelihood of seeing label k at position j as predicted by the base model. Note that we do not have to enumerate the exponentially many Y all to compute the likelihood, which can be efficiently handled by the Viterbi algorithm introduced above in linear time (Collobert et al., 2011b). Fuzzy CRF. It has been argued that properly adjusting the uncertainty of teacher prediction usually lends better knowledge transfer to the student, which can either be sharpening or relaxing the teacher predicted distributions. In standard KD, this is achieved by the incorporation of an annealing factor. In our setup, we consider relaxation via generalizing CRF to a candidate set of label sequences (Shang et al., 2018) rather than individual ones. Formally, we define the fuzzy loss as P ( Y candidate | x ) = (cid:80) y (cid:48)  Y candidate e s ( x , y (cid:48) ) (cid:80) y  Y all e s ( x , y ) , (3) where Y candidate contains candidate label sequences. Here, we will use the k -best teacher predicted label sequences as the candidate set. In this section we detail the construction of our distillation scheme, with the overall architecture outlined in Figure 3. Now we are ready to present details of our main contribution, Viterbi distillation for sequence outputs. Our basic idea is to extract information from the teacher model via drawing a set of most probable sequences, together with the respective con-fidence to those sequences. Then these sequences are presented to the student model during its training, to pass on the knowledge from teacher through various loss functions. More specifically, we apply the k -best Viterbi algorithm (see Algorithm 1 in the supplementary material) to get the pairs { ( y ( i ) 1 , p ( i, t ) 1 ) ,    , ( y ( i ) K , p ( i, t ) K ) } for sample x ( i ) , where y ( i ) j is the j -th most-likely label sequence, and p ( i, t ) j is the corresponding probability under the teacher model (indicated by the superscript t ). Similarly, in our subsequent discussions we denote p ( i, s ) j as the probability produced by the student model on the label sequence y ( i ) j . We proposed to approximate the complete cross entropy with the following k -best Vertibi approximation to avoid the exponential level of computational L Seq ce ( x ( i ) ) :=  (cid:88)  y  Y all P t ( y | x ( i ) ) log (cid:16) P s ( y | x ( i ) ) (cid:17)   K (cid:88) j =1 p ( i, t ) j log (cid:16) p ( i, s ) j ( y ( i ) j | x ( i ) ) (cid:17)  (cid:0) 1  p t sum (cid:1) log (1  p s sum ) , (4) where P t ( | ) and P s ( | ) are the CRF likelihoods respectively for teacher and student, and we collect the probability mass of the topK sequences into terms p t sum and p s sum , i.e. , Note that we have suppressed the dependency on x for notational clarity. We emphasize the residual probability mass (1  p t sum ) (cid:29) 0 (see Figure 1b), and therefore should not be excluded from the loss as in Kim and Rush (2016); Wang et al. (2020). To more appropriately account for the uncertainty in the teacher guidance, we advocate the use of a fuzzy objective that does not discriminate between the likelihood among the topK picks. Concretely, this is given by the binary cross entropy in terms of p t sum and p s sum , L Seq fuzzy ( x ( i ) ) :=  p t sum log( p s sum )  (cid:0) 1  p t sum (cid:1) log (1  p s sum ) . Multi-grained Distillation. See Figure 2 for a graphical illustration of cross entropy and fuzzy objectives, which shows different granularity on learning from teacher. In addition to the pure distillation operations, we also allow direct learning from ground-truth labels and surrogate ones labeled by the teacher model (details found in Section 3.2). These two kinds of targets are named hard labels, and the related loss is as follows, L hard ( x ( i ) ) :=  log P ( y hard | x ( i ) ) . Thus, the final loss is the weighted sum of three terms defined above, Automated Tuning of Loss Weights. Tuning the loss weights {  i } 3 i =1 in (8) via standard techniques such as grid search can be laborious and costly. In this work, we leverage the uncertainty weighting strategy proposed in (Cipolla et al., 2018) to automate the weight selection procedure to balance multi-task objectives, both for its simplicity and robust performance. More specifically, we add uncertainty regularization terms for the weights, resulting L u Seq-all = L Seq-all  1 2 (log  1 + log  2 + log  3 ) , (9) as our new learning objective. Augmenting with Unlabeled Data. NER applications are often challenged with the lack of labeled instances for training. Motivated by the observations from prior studies that data augmentation using unlabeled data may improve distillation performance (Mukherjee and Awadallah, 2019; Tang et al., 2019), we also feed unlabeled data to the teacher model to construct additional surrogate label sequences using k -best Viterbi for distillation. We pool the surrogate sequences from both labeled and unlabeled data for cross entropy and fuzzy loss. Besides, we use Top1 sequence as hard label to educate the student. Token-level Distillation. To verify the necessity of using a sequence-level distillation, we additionally consider token-level distillation with standard KD techniques. Specifically, we consider the following loss L Token :=  1 NN (cid:88) i =1 T (cid:88) j =1 L i,j , (10) where L i,j = (cid:80) Lk =1 P t ( y j = k | x ( i ) ) log( P s ( y j = k | x ( i ) )) is the token-level KL-divergence between the teacher's and student's predicted distributions for each token, rather than the token sequence distribution. This breaks the dependencies between tokens to avoid the combinatory complexity, thus enabling standard KD. The probability P ( y j = k | x ( i ) ) can be simply calculated by emission score, in this case, the loss is denoted as L Token-em . To recover the spatial dependencies, we can add transition matrix to derive the probability P ( y j = k | x ( i ) ) by calculating the marginal distribution of token from the probability distribution P ( y | x ( i ) ) given by CRF, similar to Wang et al. (2020). We denote this corresponding loss as L Token-pos . NER without BERT. Extensive investigations in NER have been conducted without appealing to the BERT architecture, and instead using simpler architectures such as BiLSTM or convolutional neural network (CNN) (Huang et al., 2015; Strubell et al., 2017a). Special effort has been made to adapt the NER network architectures for better handling of lexicons. Liu et al. (2019a) leveraged hybrid semi-Markov CRFs to improve NER recognition with Gazetteers, where segments are used as the basic units instead of words. In Zhang and Yang (2018) a lattice-structured LSTM is proposed to encode a sequence of input characters as well as all potential words that match a lexicon. Gui et al. (2019) incorporates lexicons using a rethinking mechanism under a CNN setup, which renders faster inference compared with lattice-LSTM. Peng et al. (2019) explores weighted word embedding to match all probable words given the lexicons. In the method given by Ghaddar and Langlais (2018), a lexical representation is computed for each word with a 120-dimensional vector, where element encodes the similarity of the word with an entity type. Recently, Ding et al. (2019) leveraged the graph neural network (GNN) to exploit the additional rich information captured by the gazetteers, setting new SOTA performance. Our distillation work is orthogonal to these developments, and the techniques can be combined for further improvements. Compressing BERT with Distillation. Various efforts have been made to reduce the size and cost of BERT or other big models while maximally maintaining their outstanding performance, and KD offers an appealing alternative to the direct amputation of the original models. Along this line, Tang et al. (2019) studied on compressing BERT to BiLSTM for classification, resulting a model with comparable performance to ELMo but 100  fewer parameters and 15  faster inference. Tsai et al. (2019) successfully applied KD for multilingual sequence labeling model, enabling SOTA results on an MiniBERT model afforded by a single CPU. Other developments include distillation on intermediate representations (Sun et al., 2019b) and student pre-training (Turc et al., 2019). The value of unlabeled data in KD has also been explored (Mukherjee and Awadallah, 2019). Our work complements these studies via presenting a framework of using topK soft-surrogate labels for KD. Distilling with topK Picks. Parallel to our work, Tang et al. (2020) also considered a topK scheme for KD. This work distincts from our proposal via assigning uniform weights to the un-selected labels sequences, a practice that can be largely inef-ficient. Another work close to our setup is Wang et al. (2020), where the authors tried to distill the structural knowledge from multiple monolingual teachers to a single student. In their loss, the probability mass of the non-topK picks is discarded. In Figure 1 (right), we show that the probability for non-topK picks is non-neglectable, which is properly accounted for in our proposed complete cross entropy loss. To validate the proposed solution and elaborate the gains, we benchmark it against state-of-the-art methods, through a wide range of experiments on real-world datasets. All experiments are implemented with Tensorflow and executed on a single NVIDIA P100 GPU. Details of the experimental setup are provided in supplementary material, due to space limits, and our code will be avialable at https://github.com/11zhouxuan/ multi_grained_kd_ner . The following real-world datasets are considered in our study. Detailed summary statistics of the datasets can be found in supplementary material. CoNLL-2003 NER (Sang and De Meulder, 2003) consists of newswire from Reuters RCV corpus. Unlabeled data from the Reuters RCV corpus is used for our data augmentation experiments. OntoNotes (Consortium et al., 2011) is an annotated multilingual corpus consists of texts from a wide variety of sources, such as telephone conversation, broadcast and newswire. For our NER experiment, we consider  English NER is derived from OntoNotes Release 5.0 and processed according to Pradhan et al. (2013). Chinese NER is derived from OntoNotes Release 4.0 and processed according to Che et al. (2013). Text data from other OntoNotes tasks are used as unlabeled data. MSRA (Levow, 2006) is a Chinese NER dataset with its corpus derived from news domain. We use the Chinese word segmentation dataset MSR (Levow, 2006) for unlabeled data. Weibo (Peng and Dredze, 2015) is a Chinese NER dataset derived from social media contents. Only a small fraction of data is labeled in this dataset. We briefly describe the modeling and training speci-fications choices for the teacher and student models below. Teacher Model. To build a strong learner, our teacher model is constructed by a BERT model followed by a CRF layer, and we denote it as BERT+CRF. Specifically, we use BERT BASE1 model as our feature encoder, which is known to perform strongly across a wide range of NLP tasks. A dropout layer is concatenated to the BERT, followed by a fully connected layer that computes the logit ( x ) for the labels at each location. We further apply an additional CRF-layer as defined in (2) to account for the temporal dependencies among the labels, similar to the work of Meng et al. (2019). The teacher model is trained using the standardized fine-tune paradigm (Devlin et al., 2019). Following Howard and Ruder (2018); Sun et al. (2019a), we set different learning rates for each layer. A larger rate is used for CRF, and for the BERT the learning rates decays by a factor of 0 . 9 as the layers approaches the input. Student Model. For our student model, we want it to be light, fast yet still sufficiently expressive. To this end, we use the BiLSTM+CRF architecture proposed in Huang et al. (2015). This model exploits a Bidirectional LSTM to map input sequence x into a sequence of feature vectors, which accounts for the context from both directions. The rest of the construction follows what has been described for the teacher model, with the BERT part replaced by the BiLSTM. We reuse the learned word-embeddings from the teacher model and keep it frozen during training. Empirically, we find this strategy produces better results, possibly due to reduced effort transferring the knowledge of richer embedding representation compared to alternatives such as word2vec , and it also avoids over-fitting. In addition to the vanilla teacher and student models described above, we also considered the following strong established NER baselines in our experiments. BERT teacher baselines (Chinese NER) BERT+Glyce (Meng et al., 2019). Non-BERT student baselines (English NER) LSTM-CNNs (Chiu and Nichols, 2016), LEX (Ghaddar and Langlais, 2018), IDCNN (Strubell et al., 2017b), HSCRF (Liu et al., 2019a). Non-BERT student baselines (Chinese NER) SUL (Peng et al., 2019), Lattice-LSTM (Zhang and Yang, 2018), NMDM (Ding et al., 2019), ME-CNER (Xu et al., 2019). Variations. To further understand the contributions from different components of our proposal, we run different variations of the model to dissect the gains. We use BiLSTM+CRF as our Table 1: Comparison of F1 scores. base model, and consider variants with combinations of the following components in our experiments: ( i ) TE: use fixed pre-trained teacher embedding; ( ii ) TOKEN-EM: token-level distillation with loss L Token-em ; ( iii ) TOKEN-POS: token-level distillation with with loss L Token-pos ; ( iv ) SEQ: multi-grained sequence-level distillation with loss L u Seq-all ; ( v ) DA: augmented with unlabeled data. We also vary the size of best candidate set from K = 1 to K = 15 . Evaluation. We report the F1 score following Sang and De Meulder (2003), and relegate other quantitative metrics such as Precision and Recall to the supplementary material. We apply early stopping with max patience set to 5 based on the performance of the development set. From Table 1 we first find out that, in all three Chinese datasets, our teacher model outperforms two baselines (Meng et al., 2019). We owe this to the layer-wise learning rate and discriminative fine-tuning strategies (Howard and Ruder, 2018; Sun et al., 2019a). Another analysis in terms of teacher is that directly copying the teacher embedding to the student model can be most helpful, for both English and Chinese datasets. Regarding distillation, it achieved cross-the-board performance gains relative to the no-distillation TE baseline. Our final proposal, namely TE+DA+SEQ, performs better or similarly to almost all of its non-BERT baselines. Note that these competing baselines have leveraged additional do-Table 2: Performance with different loss combinations. main knowledge, such as lexicon information, to boost NER performance (see Related Work). Such practice not only adds specialized modeling effort and complexity, but also make the resulting architecture less generalizable to other tasks. Our results show that we can match the performance via reaping the knowledge from more sophisticated models using general purpose sequence-level distillation, rather than appealing to dedicated modeling effort. We also observed that inducing data augmentation consistently improves student learning. And notably, in all cases, the sequence-level distillation outperforms token-level distillation, especially in the absence of data augmentation. Size of Candidate Set. In Figure 4, we compare how varying the size of candidate set affects performance. The performance peaks at a moderate K , after that the gain tapers off. This is because when using an excessive K , the distillation process starts to include more inaccurate label-sequences, compromising the learning efficiency (as topK label-sequences are treated equally in the fuzzy objective). Effect of Augmenting with Unlabeled Data. In Figure 5, we report the F1 scores with different number of unlabeled instances to augment the distillation phase for on Weibo dateset, which has very few training instances but an enormous unlabeled set. The use of data augmentation drastically boosts student's performance (F1: 57 . 14  71 . 62 ). The distillation performance monotonically increases as more unlabeled instances are used, and the performance closes, even beats the teacher model in the large sample limit. Loss Combinations. We further explore additional combinations of losses to sharpen our understanding, with main results summarized in Table 2. Combinations of sequence-level and token-level loss ( i.e. , L u Seq-all + L Token pos ) reveal a slight drop in performance. We also subtract each individual L Seq fuzzy and L Seq ce from L u Seq-all . It appears that the coarse-grained loss L Seq fuzzy is propitious to increase the recall value, while using the fine-grained loss L Seq ce will get better precision value. by injecting uncertainty (Cipolla et al., 2018). This method yields a regularization term of weight, preventing the weight tends to zero. Table 3 shows that, regularized weighting outperforms equal weighting in all cases. Computation Efficiency. To examine the efficiency gains, we compare inference time difference between the teacher and student model, as reported in Table 4. Our student model achieves over 40  reduction and speedup, while achieving comparable performance compared with the teacher model. Alternative Choice of Student Model. We also exam the performance of multi-grained distillation method on another student model, that is four transformer layers, which has a similarity structure as BERT. The results are showed in supplementary material (Table 9). In this work, we develop novel multi-grained knowledge distillation techniques to train a light NER model with comparable performance with their more sophisticated counterparts. In particular, we show that Viterbi algorithm can be exploited to impart the knowledge of k -best predictions from the teacher model to the student. We further advocated the use of CRF adjustments, fuzzy objective and data augmentation to improve performance. Our empirical experiments carefully analyze the gains from our proposal on a wide range of NER benchmarks, via efficiently transferring knowledge from a powerful BERT model to a much more compact BiLSTM student. In future work, we seek to extend the proposed framework to more general distillation applications where CRF is used, such as speech recognition, and distill with more general representation transfer schemes (Chen et al., 2020). The authors would like to thank the anonymous reviewers for their insightful comments. The research conducted at Duke was supported in part by the DOE, NSF and ONR.
abstain abstain objective result objective method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain objective objective other objective abstain other abstain other other other other other other other other other other objective other other other other other other other other other other other other other method other other other method method method abstain abstain method abstain other method abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain other other other other other other other other other method other other other other other other method other method abstain abstain other objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain objective result objective method abstain other other