In this paper, we study the named entity recognition (NER) problem under distant supervision. Due to the incompleteness of the external dictionaries and/or knowledge bases, such distantly annotated training data usually suffer from a high false negative rate. To this end, we formulate the Distantly Supervised NER (DS-NER) problem via Multi-class Positive and Unlabeled (MPU) learning and propose a theoretically and practically novel CONFidence-based MPU ( Conf-MPU ) approach. To handle the incomplete annotations, Conf-MPU consists of two steps. First, a confidence score is estimated for each token of being an entity token. Then, the proposed Conf-MPU risk estimation is applied to train a multi-class classifier for the NER task. Thorough experiments on two benchmark datasets labeled by various external knowledge demonstrate the superiority of the proposed Conf-MPU over existing DS-NER methods. Our code is available at Github 1 . Named Entity Recognition (NER) aims to detect entity mentions from text and classify them into predefined types. It is a fundamental task in information extraction and many other downstream tasks (Gbor et al., 2018; Luan et al., 2017; Giorgi et al., 2019). However, the necessity of extensive human efforts to annotate a large amount of training data imposes restrictions on the state-of-the-art supervised deep learning methods, especially in professional fields. To address this problem, distantly supervised methods spring up (Yang et al., 2018; Shang et al., 2018; Mayhew et al., 2019; Cao et al., 2019; Peng et al., 2019; Liang et al., 2020; Liu et al., 2021; Zhang et al., 2021), which aim to train NER models using automatically annotated training data based on external knowledge such as dictionaries and 1 https://github.com/kangISU/Conf-MPU-DS-NER Figure 1: Distant labeling example for the entity type of Disease using a dictionary. knowledge bases. Observed by previous DS-NER methods (Shang et al., 2018; Peng et al., 2019), distant labels provided by reliable dictionaries are usually of high precision. However, such distant labeling suffers a major drawback  incomplete labeling. This is due to the fact that most existing dictionaries and knowledge bases have limited coverage on entities. Hence simply treating unlabeled samples ( i.e. , unmatched tokens) as negative ones will introduce a high false negative rate ( e.g. , neutropenia in Figure 1) compared with human-annotated training data, and further mislead a supervised NER model to overfit to false negative samples and seriously affect its recall. Recently, binary Positive and Unlabeled (PU) learning is applied to DS-NER tasks for this challenge (Peng et al., 2019). PU learning performs classification using only limited labeled positive data and unlabeled data, thus naturally suitable for handling distant supervision, where external knowledge often has a limited coverage on positive samples. However, binary PU learning has several drawbacks in real DS-NER tasks. It applies the one-vs-all strategy to convert a multi-class classification problem into multiple binary classification problems, and thus suffers from two weaknesses. First, it is not efficient, especially in the case where there are many entity types. For a NER task with n entity types, n binary classifiers need to be trained. Second, the scale of predicted confidence values may differ among those binary classifiers, which may not guarantee a mutually beneficial inference for the final prediction (Bishop, 2006). Furthermore, the PU learning theory is built on a fundamental assumption of data distribution that unlabeled data can accurately reveal the overall distribution ( i.e. , the marginal distribution of the target field) (Bekker and Davis, 2020). In DS-NER tasks, the distantly annotated training data may not fit the assumption well: It depends on the coverage of used dictionaries or knowledge bases on the entities. Our empirical studies validate that violation of this assumption can significantly impact the performance of PU learning. To address these challenges in DS-NER tasks and PU learning, we propose a CONFidence-based Multi-class Positive and Unlabeled ( Conf-MPU ) learning framework. The proposed Conf-MPU can handle different levels of false negative rates brought by dictionaries of various coverage and does not overfit to the distantly labeled training data. It consists of two steps. Specifically, given the distantly labeled training data, we first carry out a token-level binary classification to estimate the confidence score (a probability value in [0 , 1] ) of a token being an entity token ( i.e. , a token of a named entity). Then, we perform the NER classification using a neural network model with the proposed Conf-MPU risk estimator, which incorporates the confidence scores obtained from the first step in the risk estimation, to alleviate the impact of annotation imperfection. It is worth noting that the two-step strategy of Conf-MPU needs to train only two classifiers for any DS-NER tasks with arbitrary number of entity types, which is more efficient than previous binary PU learning. In summary, our main contributions are:  We propose Conf-MPU, a theoretically and practically novel approach for the DS-NER task. Conf-MPU enriches the PU learning theory with solid theoretical analysis. We verify that the practical use of traditional PU learning is subject to its theoretical assumption, which can be released by Conf-MPU. As far as we know, this is the first work specially dealing with such a practical problem. We empirically demonstrate that Conf-MPU with a two-step strategy can significantly alleviate the impact of incomplete annotations during the model training and outperform the state-of-the-art DS-NER methods on benchmark datasets. In this section, we briefly review the risk formulations of standard supervised learning and PU learning in the binary classification setting. Suppose that the data follow an unknown probability distribution with density p ( x , y ) . Let x  X  R d and y  Y = { 0 , 1 } , where 0 and 1 indicate negative and positive classes, respectively. The goal is to learn a decision function f : X  Y by minimizing the expected classification risk: R( f ) =  R +P ( f ) + (1   )R  N ( f ) . In this function,  = p ( y = 1) is the prior of the positive class. R +P ( f ) = E x  p ( x | y =1) [  ( f ( x ) , 1)] and R  N ( f ) = E x  p ( x | y =0) [  ( f ( x ) , 0)] denote the expected classification risks on the positive and negative classes, respectively, where E denotes expectation and its subscript indicates the data distribution on which the expectation is computed, and the loss function is represented by  . In supervised learning setting, we are given both labeled positive and negative data that are sampled independently from p P ( x ) = p ( x | y = 1) and p N ( x ) = p ( x | y = 0) as XP = { x P j } n P j =1 and XN = { x N j } n N j =1 , respectively. Then Eq. 1 can be estimated by R PN ( f ) =  R + P ( f ) + (1   )R  N ( f ) , where R +P ( f ) = 1 n P (cid:80) n P j =1  ( f ( x P j ) , 1) and R  N ( f ) = 1 n N (cid:80) n N j =1  ( f ( x N j ) , 0) . In PU learning setting, we have only access to labeled positive data XP and unlabeled data XU = { x U j } n U j =1 drawn from p U ( x ) instead of labeled negative data XN , which indicates that the classification risk Eq. 1 can not be directly estimated as done in supervised learning setting. For this problem, Du Plessis et al. (2014) propose the expected classification risk formulation of PU learning: R( f ) =  R +P ( f ) + R  U ( f )   R  P ( f ) , (2) where R  U ( f ) = E x  p ( x ) [  ( f ( x ) , 0)] and R  P ( f ) = E x  p ( x | y =1) [  ( f ( x ) , 0)] . Here R  U ( f )   R  P ( f ) can alternatively represent (1   )R  N ( f ) because p ( y = 0) p ( x | y = 0) = p ( x )  p ( y = 1) p ( x | y = 1) . PU learning assumes that unlabeled data XU can reflect the true overall distribution, that is, p U ( x ) = p ( x ) , due to unlabeled data consisting of both positive and negative data, under which Eq. 2 can be 7199 approximated by R PU ( f ) =  R +P ( f ) + R  U ( f )   R  P ( f ) , where R  U ( f ) = 1 n U (cid:80) n U j =1  ( f ( x U j ) , 0) and R  P ( f ) = 1 n P (cid:80) n P j =1  ( f ( x P j ) , 0) . Let y  Y = { 0 , 1 , 2 , ..., k } , where 0 refers to the negative class and 1 , ..., k refer to k positive classes. The goal in multi-class classification is to minimize the following expected classification risk: R( f ) = k (cid:88) i =1  i R +P i ( f ) + (1  k (cid:88) i =1  i )R  N ( f ) , (3) where R +P i ( f ) = E x  p ( x | y = i ) [  ( f ( x ) , i )] and  i = p ( y = i ) are the classification risk and the prior of the i -th positive class, respectively. We denote this classification risk as MPN. Following PU learning setting, there are only labeled positive data XP i = { x P i j } n P i j =1 drawn from p P i ( x ) = p ( x | y = i ) where i  { 1 , ..., k } , and unlabeled data XU . Thus we can not directly estimate Eq. 3. Here we adopt the same probability princi-ple as applied in binary PU learning to alternatively compute the risk on negative data. Since p ( y = 0) p ( x | y = 0) = p ( x )  (cid:80) ki =1 p ( y = i ) p ( x | y = i ) , we can further derive Eq. 3 as: R( f ) = k (cid:88) i =1  i R +P i ( f ) + R  U ( f )  k (cid:88) i =1  i R  P i ( f ) , (4) where R  U ( f )  (cid:80) ki =1  i R  P i ( f ) theoretically plays the role of (1  (cid:80) ki =1  i )R  N ( f ) , and R  P i ( f ) = E x  p P i ( x ) [  ( f ( x ) , 0)] . Specifically, MPU risk estimator is given as: R MPU ( f ) = k (cid:88) i =1  i n P i n P i (cid:88) j =1  ( f ( x P i j ) ,i ) + max (cid:26) 0 , 1 n U n U (cid:88) j =1  ( f ( x U j ) , 0)  k (cid:88) i =1  i n P i n P i (cid:88) j =1  ( f ( x P i j ) , 0) (cid:27) , (5) with a non-negative constraint inspired by Kiryo et al. (2017) ensuring the risk on the negative class is non-negative. We denote this classification risk as MPU, whose estimation requires the same assumption of data distribution as binary PU learning does, namely, p U ( x ) = p ( x ) . We refer this assumption to PU assumption hereinafter for convenience. Under PU assumption, XU can be used to estimate R  U ( f ) . However, PU assumption can be violated in real distant supervision scenarios. The distribution of unlabeled data p U ( x ) may be different from the overall distribution p ( x ) especially when the distant supervision has a good coverage. In such cases, the unlabeled data will have a distribution closer to the distribution of the true negative data p N ( x ) instead of the overall distribution p ( x ) . Thus, the risk estimation of R  U ( f ) based on the assumption, in either MPU or binary PU, may be biased. To alleviate such estimation bias, we derive a novel Conf-MPU risk function from MPU. With the context of NER tasks, we observe that almost any combination of characters could be part of a named entity. Based on this observation, mathematically, we define  ( x ) = p ( y > 0 | x ) to determine the confidence score of a token being an entity token, no matter what entity type it belongs to, and further assume that  ( x ) > 0 . Under this assumption, we can further decompose R  U ( f ) in Eq. 4 by involving a threshold parameter 0 <   1 as follows: R  U ( f ) = k (cid:88) i =1  i R   P i ( f ) + R   U ( f ) , (6) where R   P i ( f ) = E x  p P i ( x |  ( x ) > ) (cid:104)  ( f ( x ) , 0) 1  ( x ) (cid:105) and R   U ( f ) = E x  p ( x |  ( x )   ) [  ( f ( x ) , 0)] . The detailed proof is shown as follows. Proof. Since  ( x ) > 0 and 0 <   1 , we have R  U ( f ) = E x  p ( x ) [  ( f ( x ) , 0)] = (cid:90)  ( x ) >  ( f ( x ) , 0) p ( x ) d x + (cid:90)  ( x )    ( f ( x ) , 0) p ( x ) d x = (cid:90)  ( x ) >  ( f ( x ) , 0) p ( x ) p ( x ,y > 0) p ( x ,y > 0) d x + R   U ( f ) = (cid:90)  ( x ) >  ( f ( x ) , 0) p ( x ) p ( x | y > 0) p ( y > 0) p ( y > 0 | x ) p ( x ) d x + R   U ( f ) = (cid:90)  ( x ) >  ( f ( x ) , 0) p ( x | y > 0) p ( y > 0)  ( x ) d x + R   U ( f ) = k (cid:88) i =1 p ( y = i ) (cid:90)  ( x ) >  ( f ( x ) , 0) p ( x | y = i )  ( x ) d x + R   U ( f ) = k (cid:88) i =1  i (cid:90)  ( x ) >  ( f ( x ) , 0) 1  ( x ) p ( x | y = i ) d x + R   U ( f ) = k (cid:88) i =1  i R   P i ( f ) + R   U ( f ) . Given a reliable  and a proper  ,  ( x ) >  indicates x being an entity token (a positive sample), otherwise a non-entity token (a negative sample), which further induces that p P i ( x |  ( x ) >  )  p P i ( x ) , and p ( x |  ( x )   )  p U ( x |  ( x )   )  p N ( x ) even if p U ( x ) is different from p ( x ) Thus, empirically R   P i ( f ) and R   U ( f ) can be estimated with less bias using XP i and XU , respectively, which further leads to a more precise estimation of R  U ( f ) . This is the mechanism that Conf-MPU can significantly reduce estimation bias in practice, even if PU assumption is violated. Specifically, Conf-MPU risk estimator can be expressed as: R Conf  MPU ( f ) = k (cid:88) i =1  i n P i n P i (cid:88) j =1 max (cid:26) 0 ,  ( f ( x P i j ) , i ) + 1   ( x P i j ) >  ( f ( x P i j ) , 0) 1   ( x P i j )   ( f ( x P i j ) , 0) (cid:27) + 1 n U n U (cid:88) j =1 (cid:104) 1   ( x U j )    ( f ( x U j ) , 0) (cid:105) , (8) with a constraint to guarantee a non-negative loss on each labeled positive sample, where   is an empirical confidence score estimator. . In DS-NER tasks, we formulate the sub-task of estimating  ( x ) as a token-level binary classification problem which also uses distant labels. In practice, a classifier with a sigmoid output layer for this sub-task can guarantee   ( x ) > 0 . Targeting on the challenge of high false negative rates in training data, we give the following analysis to offer some insights into the Conf-MPU risk estimator. For ease of expression, we use letters to denote the terms in Eq. (8): A =  ( f ( x P i j ) , i ) , B = 1   ( x P i j ) >  ( f ( x P i j ) , 0) 1   ( x P i j ) , C =  ( f ( x P i j ) , 0) , D = 1   ( x U j )    ( f ( x U j ) , 0) . The threshold  is set to 0.5 by default. We assume that   ( x ) of an entity token is close to 1 ( i.e. ,   ( x ) >  ), and   ( x ) of a non-entity token is close to 0 ( i.e. ,   ( x )   ). For a true positive sample ( e.g. , sepsis distantly labeled in Figure 1), the loss is computed by A + B  C , where B is involved because its confidence score is larger than the threshold. Since 1 /   ( x ) is close to 1, B  C is almost 0 but positive, and thus the loss on this sample approximately equals to A , which is very similar with the loss on a positive sample in standard supervised learning. For a true negative sample ( e.g. , patient unlabeled in Figure 1), the loss is calculated by D due to its confidence score is less than the threshold. So the minimization for D enables the model to learn from this true negative sample. For a false negative sample ( e.g. , neutropenia unlabeled in Figure 1), the loss is not counted, because its confidence score is larger than the threshold and thus D is not calculated. It is the mechanism that Conf-MPU handles false negative samples from unlabeled data. Here we establish an estimation error bound for the proposed Conf-MPU risk estimator (Eq. 8) to show the guaranteed performance. Theorem 1. Let f  = arg min f F R( f ) and  f Conf  MPU = arg min f F R Conf  MPU ( f ) . Assume that  (  )  [0 , C l ] and  is Lipschitz continuous on the interval [  C g , C g ] with a Lipschitz constant L l , where C l , C g > 0 . Also suppose that   is a fixed function independent of data used to compute R Conf  MPU ( f ) and   (0 , 1] . Let  = p (  ( x )   ) and  = E x  p ( x ) (cid:104) |   ( x )   ( x ) | 2 (cid:105) . Then for any  > 0 , with probability at least 1   , R(  f Conf  MPU )  R( f  )  k (cid:88) i =1 2  i (  + 1) C l  + k (cid:88) i =1 2  i   2 L l  R n P i ,p P i ( x ) ( F ) + (  + 1) C l  (cid:115) log k +1  2 n P i   + 4 L l R n U ,p ( x ) ( F ) + 2 C l (cid:115) log k +1  2 n U + 2 C l  (cid:112) (1   ) . In Theorem 1, F is the function class and R n P i ,p P i ( x ) ( F ) is the Rademacher complexity of the function class F for the sampling of size n P i from the distribution p P i ( x ) and R n U ,p ( x ) ( F ) follows a similar definition. We relegate this proof to the Appendix. In this section, we describe the setup for the DS-NER classification. In DS-NER tasks, professional dictionaries ( e.g. , UMLS) and knowledge bases ( e.g. , Wikidata) are used to automatically generate distant labels. Distant labeling by dictionaries employs some string matching algorithms to map training samples to dictionaries (Ren et al., 2015; Giannakopoulos et al., 2017; Peng et al., 2019), while knowledge bases utilize public APIs to perform such distant labeling. The proposed Conf-MPU risk estimator can be applied on any NER classifiers where the task is to predict the label for each token. For example, 7201 BERT (Devlin et al., 2019) can be used as the underlying NER model, and then the Conf-MPU risk estimation can be used to calculate the classification risks. We use Conf-MPU BERT to denote this method. BiLSTM (Chiu and Nichols, 2016) is another popular choice for NER models. Ratinov and Roth (2009); Passos et al. (2014); Chiu and Nichols (2016) demonstrate that using lexicons as external features can improve NER performance. With the dictionaries, we extract the lexicon features as follows. For each token, we match its contextual words within a window size against entries in the dictionaries. If there is any successful matching, a binary indicator is set to 1, otherwise 0. With the window size of n , we can form an n -bit vector, which is appended to the input embedding. We denote the model of BiLSTM with the lexicon feature engineering as LBiLSTM, and denote Conf-MPU LBiLSTM as the LBiLSTM-based classifier with Conf-MPU risk estimation. For the first step of estimating confidence scores, we build a token-level binary classifier ( i.e. ,  ) based on LBiLSTM to output scores. To be consistent with PU learning setting, this classifier is equipped with a binary PU learning risk estimator ( i.e. , R PU (  ) ). Unlike in supervised learning where priors ( i.e. ,  i ) can be easily obtained from human annotations, we cannot directly acquire them from distant annotations. In PU learning research, there are some methods proposed specifically for estimating the priors (Bekker and Davis, 2018; Jain et al., 2016; Du Plessis and Sugiyama, 2014). Here we adopt the most effective TIcE algorithm from Bekker and Davis (2018) to perform prior estimation. Peng et al. (2019) point out that a bounded loss function can help avoid overfitting in PU learning setting. We also confirm this argument in our empirical studies. Thus, instead of using the common unbounded cross entropy loss function, we adopt the mean absolute error (MAE) as the loss function for Conf-MPU and other PU learning methods in our experiments. Given its label y in the one-hot form, the loss on a token x is defined by:  ( f ( x ) , y ) = 1 k + 1 k (cid:88) i =0 | y ( i )  f ( x ) ( i ) | , where f ( x ) is the softmax output, and both y and f ( x ) are in k +1 dimensions. Note that  ( f ( x ) , y )  [0 , 2 k +1 ] is bounded. In DS-NER tasks, self-training strategies as postprocessing can often further improve the performance, such as iteratively enriching dictionaries based on the model predictions (Peng et al., 2019), or iteratively training a teacher-student framework (Liang et al., 2020). The discussion for self-training framework is out of the scope of this paper and we refer the readers to Zoph et al. (2020) for more information. We consider two benchmark NER datasets from different domains: (1) BC5CDR comes from biomedical domain. It consists of 1,500 articles, containing 15,935 Chemical and 12,852 Disease mentions; (2) CoNLL2003 is a well-known open-domain NER dataset. It consists of 1,393 English news articles, containing 10,059 PER , 10,645 LOC , 9,323 ORG and 5,062 MISC mentions. We obtain the following distantly labeled datasets: (1) BC5CDR (Big Dict) is labeled using a dictionary 2 released by Shang et al. (2018); (2) BC5CDR (Small Dict) is labeled using a smaller dictionary constructed by selecting only the first 20% entries from the previous one; (3) CoNLL2003 (KB) 3 is labeled by the knowledge base Wikidata and released by Liang et al. (2020); (4) CoNLL2003 (Dict) is labeled using a refined dictionary released by Peng et al. (2019) 4 . For dictionary labeling, we use the strict string matching algorithm presented in Peng et al. (2019). The process of knowledge base labeling can be found in Liang et al. (2020). All DS-NER methods are trained on the same distantly labeled training data and evaluated on the released human-annotated test sets in terms of span-level precision, recall and F1 score. To avoid the noise induced by the position tag in the distant labels, we do not consider the position of 2 https://github.com/shangjingbo1226/AutoNER 3 https://github.com/cliang1453/BOND 4 https://github.com/v-mipeng/LexiconNER 7202 each token in a named entity. During the prediction phase, a continuous span with the same label is considered as a single entity. We compare the proposed Conf-MPU with different groups of baseline methods. Fully Supervised Methods. We present the state-of-the-art (SOTA) performance of fully supervised methods on the two benchmark datasets, Wang et al. (2021) on BC5CDR and Wang et al. (2020) on CoNLL2003. For SOTA methods, we report the results from their original papers. We also evaluate the employed BiLSTM and BERT models in fully supervised setting. The performance in this group serves as upper-bound references. Distantly Supervised Methods. We consider the following distantly supervised NER methods: (1) Dict/KB Matching distantly labels the test sets using dictionaries or knowledge bases directly, which is included here as references; (2) AutoNER (Shang et al., 2018) trains the model using a tie-or-break mechanism to detect entity boundaries and then predicts entity type for each candidate; (3) BERT-ES (Liang et al., 2020) adopts early stopping to prevent BERT from overfitting to noisy distant labels; (4) BNPU (Peng et al., 2019) built on LBiLSTM ( BNPU LBiLSTM ) applies a binary PU learning risk estimation with MAE as the loss function to each entity type and then infers the final types; (5) MPU is the predecessor of the proposed Conf-MPU, which computes the empirical risk using Eq. 5. We also build MPU on both BERT and LBiLSTM models, denoted as MPUBERT and MPU LBiLSTM . Note that full models in Peng et al. (2019); Liang et al. (2020) contain self-training as post processing steps, which are omitted here. We focus on the evaluation of how well each model can handle incomplete labeling issues in DS-NER tasks. To evaluate the efficacy of the DS-NER methods in real usage under distantly supervised settings, we do not use any human-annotated validation or test sets in any stage of the training process. The training stopping criteria are set as follows: 100 epochs for BiLSTM-based methods and 5 epochs for BERT-based ones. We report the performance of the final model instead of the best checkpoint. Consequently, the baselines have different performance from their re-Dataset Type Precision Recall BC5CDR (Big Dict) Chemical 97.99 63.14 Disease 98.34 46.73 BC5CDR (Small Dict) Chemical 98.66 11.43 Disease 99.25 9.31 CoNLL2003 (KB) PER 82.36 82.11 LOC 99.98 65.20 ORG 90.47 60.59 MISC 100.00 20.07 CoNLL2003 (Dict) PER 99.78 79.10 LOC 97.56 34.69 ORG 95.80 65.47 MISC 99.24 57.22 Table 1: The quality of distant labels on training sets, stated in token-level precision and recall (in %). ported results. We use the released code for AutoNER and BERT-ES to reproduce their results. For other methods, we report the results based on our implementations. BiLSTM-based models utilize pretrained bio-embedding 5 for BC5CDR and pretrained Stanford's Glove 6 embedding for CoNLL2003. BERT-based models use pretrained biobert-base-cased-v1.1 7 for BC5CDR and bert-base-cased 8 for CoNLL2003. The only exception is that BERT-ES uses roberta-base 9 for CoNLL2003 in the original implementation. We first examine the quality of the distantly labeled training data. Table 1 shows the detailed evaluation of distantly labeled training data. The results validate the assumption mentioned in previous work that distant labels generated by dictionaries are often of high precision but low recall. Table 2 presents the overall span-level precision, recall, and F1 scores for all methods on the test sets. The proposed Conf-MPU shows a clear advantage over baseline methods, especially when accompanying with LBiLSTM. Almost all distantly supervised baselines perform better than Dict/KB Matching on these four datasets, except for a few cases of BNPU and MPU which will be discussed later. Among the baseline methods, AutoNER and BERT-ES show strong correlation with respect to the dictionary quality. On BC5CDR (Small Dict), where the dictionary suffers from extremely low coverage, the two methods have little improvement 5 https://github.com/shangjingbo1226/AutoNER 6 https://nlp.stanford.edu/projects/glove/ 7 https://huggingface.co/dmis-lab/biobert-base-cased-v1.1 8 https://huggingface.co/bert-base-cased 9 https://huggingface.co/roberta-base 7203 Method BC5CDR (Big Dict) BC5CDR (Small Dict) CoNLL2003 (KB) CoNLL2003 (Dict) Fully Supervised Existing SOTA 90.99 (-/-) 94.60 (-/-) BERT 83.88 (79.75/88.46) 89.03 (88.00/90.08) BiLSTM 75.60 (71.27/80.49) 86.19 (84.06/88.42) Distantly Supervised Dict/KB Matching 64.32 (86.39/51.24) 15.69 (80.02/8.70) 71.40 (81.13/63.75) 63.93 (93.12/48.67) AutoNER 79.99 (82.63/77.52) 20.66 (81.47/11.83) 67.80 (73.10/63.22) 61.19 (82.87/48.50) BERT-ES 73.66 (80.43/67.94) 17.21 (75.60/9.71) 72.15 (81.38/64.80) 63.68 (85.77/50.63) BNPU LBiLSTM 59.24 (48.12/77.06) 70.21 (64.93/76.43) 78.44 (74.38/82.97) 76.11 (73.68/78.70) MPUBERT 68.22 (56.50/86.05) 73.91 (70.08/78.18) 65.75 (58.79/74.58) 67.65 (63.63/72.22) MPU LBiLSTM 60.79 (48.28/82.06) 73.25 (67.50/80.07) 69.13 (59.46/82.54) 71.41 (63.41/81.71) Conf-MPU BERT 77.22 (69.79/86.42) 71.85 (81.02/64.54) 79.16 (78.58/79.75) 81.89 (81.71/82.08) Conf-MPU LBiLSTM 80.07 (76.63/83.82) 76.18 (82.66/70.64) 80.02 (77.39/82.84) 83.34 (85.79/81.02) Table 2: The span-level results on test sets: F1 score (Precision/Recall) (in %), where the bests are in bold. on recall. Similarly, this phenomenon can also be observed by comparing their performance on CoNLL2003 (KB) and CoNLL2003 (Dict). By contrast, all PU learning based methods demonstrate significantly higher recall on all datasets, showing more robustness to the issue of incomplete labeling. However, we can observe that compared with their performances on BC5CDR (Small Dict), BNPU and MPU suffer from low precision on BC5CDR (Big Dict) labeled by a dictionary with high coverage and precision, which is against one's intuition. We will extend the discussion in Section 5.2.3. Note that although BNPU and MPU are derived from the same probability princi-ple, it is not necessary for BNPU and MPU to have similar performance. For example, BNPU LBiLSTM and MPU LBiLSTM perform similarly on BC5CDR datasets, but not on CoNLL2003 datasets. We suspect the cause is that they differ in the training process. BNPU is trained with the one-vs-all strategy where the distribution of unlabeled data is different for each entity type, while MPU is simultaneously trained with all types keeping the same distribution of unlabeled data. As mentioned earlier, the distribution of unlabeled data may significantly affect the risk estimation of PU learning. In addition, the inference step also has an unpredictable effect on the overall performance of BNPU. As the results manifest, Conf-MPU can significantly improve precision compared with BNPU and MPU, and meanwhile maintain a high level of recall on all datasets, which shows that Conf-MPU can significantly alleviate the estimation bias. For Conf-MPU, another factor for its performance is the confidence score estimation for each token being an entity token. To evaluate the quality of the confidence scores, we first convert the results as labels where if   ( x ) > 0 . 5 then label x as an 7204 Method BC5CDR (Big Dict) BC5CDR (Small Dict) Fully Supervised 85.00 (77.68/93.83) Binary PU 72.38 (61.89/87.17) 79.14 (81.85/76.60) Method CoNLL2003 (KB) CoNLL2003 (Dict) Fully Supervised 96.09 (92.77/99.65) Binary PU 88.88 (81.04/98.39) 88.08 (79.09/99.38) Table 3: The results of confidence score estimation on test sets: F1 score (Precision/Recall) (in %). entity token, otherwise label as a non-entity token. We present the results in terms of token-level F1 score, precision, and recall in Table 3, where Fully Supervised using human-annotated ground-truth labels provides upper-bound references for this estimation, while Binary PU uses distant labels. We can see that the classifier with the binary PU risk estimation achieves good recall on all of the distantly labeled datasets. High recall indicates that the classifier can recognize most of entity tokens, which can be taken advantage of in the Conf-MPU risk estimation to avoid overfitting to false negative samples in unlabeled data. We also evaluate the proposed Conf-MPU models with the confidence scores given by the fully supervised classifier on the four distantly labeled datasets, where the performances increased by 2  5 percentage in terms of F1 score, indicating that the proposed Conf-MPU framework is robust to the confidence score estimation of lesser quality. We leave for future work the optimization of the confidence score estimation. To have a solid recognition to the estimation bias in BNPU and MPU caused by the violation of PU assumption, we construct a series of dictionaries with different coverage on entities. We treat the dictionaries used to generate labels for BC5CDR (Big Dict) and CoNLL2003 (Dict) as two reference standard dictionaries. Then for each of the two benchmark datasets we build a group of dictionaries by selecting the first 20, 40, 60, 80, and 100 (%) entries from the standard ones. We train BNPU, MPU, and Conf-MPU on the distantly labeled datasets generated by these dictionaries. Here we show the results based on LBiLSTM in Figure 2 ( a c and f h on corresponding test sets). Similar trend can be observed on BERT-based settings. We can see a clear decreasing trend on precision for BNPU and MPU when dictionary size increases (Figures 2 ( a&f )). These phenomena are caused due to the violation of PU assumption. When a dictionary has higher coverage, the distribution of unlabeled data is more and more similar to the Method LBiLSTM BiLSTM BNPU 70.21 (64.93/76.43) 63.37 (57.92/69.97) MPU 73.25 (67.50/80.07) 62.39 (56.50/69.66) Conf-MPU 76.18 (82.66/70.64) 68.11 (71.68/64.88) Table 4: Ablation study on lexicon features on BC5CDR (Small Dict). distribution of true negative data, instead of to the overall data distribution. The BNPU and MPU risk estimations bring higher bias, leading to lower precision. Although their recalls remain high, the F1 scores still decrease. By contrast, the proposed Conf-MPU can effectively avoid this limitation and achieve good performance for all dictionary sizes. To further evaluate the Conf-MPU risk estimation, we first conduct ablation studies comparing with MPN risk (Eq. 3) whose estimation simply treats unlabeled data as negative samples in distant supervision, and demonstrate the performance with different epochs. Sub-figures ( d , e , i , j ) in Figure 2 show the trends of F1 scores of LBiLSTM-based models on the validation sets using three risk estimations of Conf-MPU, MPU, and MPN, with respect to different number of epochs. ( d , e ) and ( i , j ) reflect the performance on BC5CDR and CoNLL2003, respectively. ( d , i ) and ( e , j ) reflect the performance based on 20% and 100% dictionaries, respectively. The results show that MPN risk estimation can lead to severe overfitting for the model when dictionaries have low coverage. Although MPN still causes overfitting on full dictionaries, its performances are more stable and generally good. By contrast, MPU and Conf-MPU consider the false negative issue during training, and do not overfit even on small dictionaries. Conf-MPU performs stably and consistently well for more scenarios comparing with the other risk estimations. From Table 2, we can observe that Conf-MPU LBiLSTM outperforms BiLSTM with fully supervised setting on BC5CDR with both big and small dictionaries. To examine the performance gain, we implement three methods, BNPU, MPU, and Conf-MPU based on BiLSTM instead of LBiLSTM to evaluate the impact of lexicon features learned from the dictionaries. The results on BC5CDR (Small Dict) are shown in Table 4. We can see that the lexicon features used in DS-NER tasks can significantly improve the performance. The experiments performed on other distantly labeled datasets also exhibit similar trends. The results suggest that dic-7205 tionaries in DS-NER tasks can also serve as external features in additional to the distant labels. DS-NER. Handling noisy labels (false positives and false negatives) in DS-NER has attracted extensive attention in recent years (Yang et al., 2018; Shang et al., 2018; Mayhew et al., 2019; Cao et al., 2019; Peng et al., 2019; Liang et al., 2020; Liu et al., 2021; Zhang et al., 2021). Here we briefly discuss a few representative approaches. One line of work focuses on alleviating the impact of false negatives (or incomplete labeling). Our work belongs to this line. AutoNER (Shang et al., 2018) proposes a new tagging scheme to identify entity candidates by determining if the connection of two adjacent tokens should be tied , broken , or unknown , and then decides the type for entity candidates. To handle incomplete labeling, tokens with unknown tag are not counted for the loss calculation. Mayhew et al. (2019) introduce a constraint driven iterative algorithm learning to detect false negatives in the noisy data and down-weigh them, resulting in a weighted training set on which a weighted NER model is trained. Peng et al. (2019) employ PU learning to avoid the model overffiting to false negatives, and propose a bounded non-negative positive-unlabeled learning. However, the application of binary PU learning in DS-NER is limited to the underlying PU assumption and its efficiency. Our proposed Conf-MPU can release the limitations and allow PU learning to be utilized in wider distant supervision scenarios. Another line of work considers the noise of both types, either explicitly or implicitly. Cao et al. (2019) design a data selection scheme to compute scores for annotation confidence and annotation coverage to distinguish high-quality sentences from noisy ones, and then propose a name tagging model that consists of two modules of sequence labeling and classification, focusing on high-quality and noisy portions, respectively. BOND (Liang et al., 2020), leveraging the power of pre-trained language model BERT, first adopts early stopping to prevent overfitting to noisy labels and obtains an initialized model, then further boosts the performance by a teacher-student self-training framework. Liu et al. (2021) propose a calibrated confidence estimation approach for DS-NER and integrate it in an LSTM-CRF model under a self-training framework to reduce the impact of noise. Zhang et al. (2021) study the noise in DS-NER from a novel perspective of dictionary bias. Specifically, they first formulate DS-NER using a structural causal model, then identify the causes of both false positives and false negatives, and finally de-bias via backdoor adjustment and causal invariance regularizer. We leave for future work enabling Conf-MPU to handle false positives. PU Learning. PU learning learns a classifier from positive and unlabeled data (Elkan and Noto, 2008; Du Plessis et al., 2014). In a broad sense, PU learning belongs to semi-supervised learning. However, there is a fundamental difference between them: semi-supervised learning requires labeled negative data, but PU learning does not. Recently, a few works significantly enriched PU learning theory. Kiryo et al. (2017) propose a non-negative risk estimator for PU learning, which enables the usage of deep neural networks for classification given limited labeled positive data. Xu et al. (2017) first come up with the concept of multi-positive and unlabeled learning with a margin maximization goal for the multi-class classification problem. However, the objective of margin maximization cannot be easily extended to apply on popular deep learning architectures. Hsieh et al. (2019) propose a novel classification framework incorporating biased negative data in PU learning, which opens up a wider range of the applications of PU learning. In this paper, we present a novel multi-class positive and unlabeled learning method called Conf-MPU for the DS-NER task. Conf-MPU estimates the empirical classification risks using the confidence estimation of a token being an entity token on the distantly labeled training data to prevent the model from overfitting to the false negatives. We empirically show that Conf-MPU can significantly reduce the potential risk estimation bias caused by PU assumption. The extensive experiments illustrate that compared with existing DS-NER methods, Conf-MPU is more robust to various types of dictionaries and can handle the incomplete labeling problem effectively. The work is supported in part by NIFA grant no. 2022-67015-36217 from the USDA National Institute of Food and Agriculture.
method abstain objective abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective objective abstain objective abstain result objective objective abstain abstain abstain abstain abstain other method abstain other abstain abstain other abstain abstain other abstain other abstain abstain abstain method abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain other abstain abstain abstain abstain other abstain method abstain method method other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method abstain method method other abstain abstain abstain abstain abstain abstain other abstain abstain abstain other abstain method abstain method result method abstain abstain method abstain method abstain method method abstain method other abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain result result abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain other other abstain other abstain other other other other other objective other other other other other other abstain other other other other other other other other other objective abstain result abstain other