We study the problem of learning a named entity recognition (NER) tagger using noisy labels from multiple weak supervision sources. Though cheap to obtain, the labels from weak supervision sources are often incomplete, inaccurate, and contradictory, making it difficult to learn an accurate NER model. To address this challenge, we propose a conditional hidden Markov model (CHMM), which can effectively infer true labels from multi-source noisy labels in an unsupervised way. CHMM enhances the classic hidden Markov model with the contextual representation power of pretrained language models. Specifically, CHMM learns token-wise transition and emission probabilities from the BERT embeddings of the input tokens to infer the latent true labels from noisy observations. We further refine CHMM with an alternate-training approach (CHMM-ALT). It fine-tunes a BERT-NER model with the labels inferred by CHMM, and this BERT-NER's output is regarded as an additional weak source to train the CHMM in return. Experiments on four NER benchmarks from various domains show that our method outperforms state-of-the-art weakly supervised NER models by wide margins. Named entity recognition (NER), which aims to identify named entities from unstructured text, is an information extraction task fundamental to many downstream applications such as event detection (Li et al., 2012), relationship extraction (Bach and Badaskar, 2007), and question answering (Khalid et al., 2008). Existing NER models are typically supervised by a large number of training sequences, each pre-annotated with token-level labels. In practice, however, obtaining such labels could be prohibitively expensive. On the other hand, many domains have various knowledge resources such as knowledge bases, domain-specific dictionaries, or labeling rules provided by domain experts (Far-makiotou et al., 2000; Nadeau and Sekine, 2007). These resources can be used to match a corpus and quickly create large-scale noisy training data for NER from multiple views. Learning an NER model from multiple weak supervision sources is a challenging problem. While there are works on distantly supervised NER that use only knowledge bases as weak supervision (Mintz et al., 2009; Shang et al., 2018; Cao et al., 2019; Liang et al., 2020), they cannot leverage complementary information from multiple annotation sources. To handle multi-source weak supervision, several recent works (Nguyen et al., 2017; Safranchik et al., 2020; Lison et al., 2020) leverage the hidden Markov model (HMM), by modeling true labels as hidden variables and inferring them from the observed noisy labels through unsupervised learning. Though principled, these models fall short in capturing token semantics and context information, as they either model input tokens as one-hot observations (Nguyen et al., 2017) or do not model them at all (Safranchik et al., 2020; Lison et al., 2020). Moreover, the flexibility of HMM is limited as its transitions and emissions remain constant over time steps, whereas in practice they should depend on the input words. We propose the conditional hidden Markov model (CHMM) to infer true NER labels from multi-source weak annotations. CHMM conditions the HMM training and inference on BERT by predicting token-wise transition and emission probabilities from the BERT embeddings. These token-wise probabilities are more flexible than HMM's constant counterpart in modeling how the true labels should evolve according to the input tokens. The context representation ability they inherit from BERT also relieves the Markov constraint and expands HMM's context-awareness. Further, we integrate CHMM with a supervised BERT-based NER mode with an alternate-training method (CHMM-ALT). It fine-tunes BERT-NER with the denoised labels generated by CHMM. Taking advantage of the pre-trained knowledge contained in BERT, this process aims to refine the denoised labels by discovering the entity patterns neglected by all of the weak sources. The fine-tuned BERT-NER serves as an additional supervision source, whose output is combined with other weak labels for the next round of CHMM training. CHMM-ALT trains CHMM and BERT-NER alternately until the result is optimized. Our contributions include:  A multi-source label aggregator CHMM with token-wise transition and emission probabilities for aggregating multiple sets of NER labels from different weak labeling sources. An alternate-training method CHMM-ALT that trains CHMM and BERT-NER in turn utilizing each other's outputs for multiple loops to optimize the multi-source weakly supervised NER performance. A comprehensive evaluation on four NER benchmarks from different domains demonstrates that CHMM-ALT achieves a 4 . 83 average F1 score improvement over the strongest baseline models. The code and data used in this work are available at github.com/Yinghao-Li/CHMM-ALT. Weakly Supervised NER There have been works that train NER models with different weak supervision approaches. Distant supervision , a spe-cific type of weak supervision, generates training labels from knowledge bases (Mintz et al., 2009; Yang et al., 2018; Shang et al., 2018; Cao et al., 2019; Liang et al., 2020). But such a method is limited to one source and falls short of acquiring supplementary annotations from other available resources. Other works adopt multiple additional labeling sources, such as heuristic functions that depend on lexical features, word patterns, or document information (Nadeau and Sekine, 2007; Rat-ner et al., 2016), and unify their results through multi-source label denoising . Several multi-source weakly supervised learning approaches are designed for sentence classification (Ratner et al., 2017, 2019; Ren et al., 2020; Yu et al., 2020). Although these methods can be adapted for sequence labeling tasks such as NER, they tend to overlook the internal dependency relationship between token-level labels during the inference. Fries et al. (2017) target the NER task, but their method first generates candidate named entity spans and then classifies each span independently. This inde-pendence makes it suffer from the same drawback as sentence classification models. A few works consider label dependency while dealing with multiple supervision sources. Lan et al. (2020) train a BiLSTM-CRF network (Huang et al., 2015) with multiple parallel CRF layers, each for an individual labeling source, and aggregate their transitions with confidence scores predicted by an attention network (Bahdanau et al., 2015; Lu-ong et al., 2015). HMM is a more principled model for multi-source sequential label denoising as the true labels are implicitly inferred through unsupervised learning without deliberately assigning any additional scores. Following this track, Nguyen et al. (2017) and Lison et al. (2020) use a standard HMM with multiple observed variables, each from one labeling source. Safranchik et al. (2020) propose linked HMM, which differs from ordinary HMM by introducing unique linking rules as an adjunct supervision source additional to general token labels. However, these methods fail to utilize the context information embedded in the tokens as effectively as CHMM, and their NER performance is further constrained by the Markov assumption. Neuralizing the Hidden Markov Model Some works attempt to neuralize HMM in order to relax the Markov assumption while maintaining its generative property (Kim et al., 2018). For example, Dai et al. (2017) and Liu et al. (2018) incorporate recurrent units into the hidden semi-Markov model (HSMM) to segment and label high-dimensional time series; Wiseman et al. (2018) learn discrete template structures for conditional text generation using neuralized HSMM. Wessels and Omlin (2000) and Chiu and Rush (2020) factorize HMM with neural networks to scale it and improve its sequence modeling capacity. The work most related to ours leverages neural HMM for sequence labeling (Tran et al., 2016). CHMM differs from neural HMM in that the tokens are treated as a dependency term in CHMM instead of the observation in neural HMM. Besides, CHMM is trained with generalized EM, whereas neural HMM opti-Center in New York was... mizes the marginal likelihood of the observations. In this section, we formulate the multi-source weakly supervised NER problem. Consider an input sentence that contains T tokens w (1: T ) , NER can be formulated as a sequence labeling task that assigns a label to each token in the sentence. 1 Assuming the set of target entity types is E and the tagging scheme is BIO (Ramshaw and Marcus, 1995), NER models assign one label from the label set l  L to each token, where the size of the label set is |L| = 2 |E| + 1 , e.g. , if E = { PER , LOC } , then L = { O , B-PER , I-PER , B-LOC , I-LOC } . Suppose we have a sequence with K weak sources, each of which can be a heuristic rule, knowledge base, or existing out-of-domain NER model. Each source serves as a labeling function that generates token-level weak labels from the input corpus, as shown in Figure 1. For the input sequence w (1: T ) , we use x (1: T ) k , k  { 1 , . . . , K } to represent the weak labels from the source k , where x ( t ) k  R |L| , t  { 1 , . . . , T } is a probability distribution over L . Multi-source weakly supervised NER aims to find the underlying true sequence of labels  y (1: T ) ,  y ( t )  L given { w (1: T ) , x (1: T ) 1: K } . In this section, we describe our proposed method CHMM-ALT. We first sketch the alternate-training procedure ( 4.1), then explain the CHMM component ( 4.2) and how BERT-NER is involved ( 4.3). The alternate-training method trains two models a multi-source label aggregator CHMM and a BERT-NER modelin turn with each other's output. CHMM aggregates multiple sets of labels from different sources into a unified sequence of 1 We represent vectors, matrices or tensors with bold fonts and scalars with regular fonts; 1 : a (cid:44) { 1 , 2 , . . . , a } . In phase I , CHMM takes the annotations x (1: T ) 1: K from existing sources and gives a set of denoised labels y  (1: T ) , which are used to fine-tune the BERT-NER model. Then, we regard the fine-tuned model as an additional labeling source, whose outputs  y (1: T ) are added into the original weak label sets to give the updated observation instances: x (1: T ) 1: K +1 = { x (1: T ) 1: K ,  y (1: T ) } . In phase II , CHMM and BERT-NER mutually improve each other iteratively in several loops. Each loop first trains CHMM with the observation x (1: T ) 1: K +1 from the previous one. Then, its predictions are adopted to fine-tune BERT-NER, whose output updates x (1: T ) K +1 . Figure 2 illustrates the alternate-training method. In general, CHMM gives high precision predictions, whereas BERT-NER trades recall with precision. In other words, CHMM can classify named entities with high accuracy but is slightly disadvantaged in discovering all entities. BERT-NER increases the coverage with a certain loss of accuracy. Combined with the alternate-training approach, this complementarity between these models further increases the overall performance. The conditional hidden Markov model is an HMM variant for multi-source label denoising. It models true entity labels as hidden variables and infers them from the observed noisy labels. Traditionally, discrete HMM uses one transition matrix to model the probability of hidden label transitioning and one emission matrix to model the probability of the observations from the hidden labels. These two matrices are constant, i.e. , their values do not change over time steps. CHMM, on the contrary, conditions both its transition and emission matrices on the BERT embeddings e (1: T ) of the input tokens w (1: T ) . This design not only allows CHMM to leverage the rich contextual representations of the BERT embeddings but relieves the constant matrices constraint as well. In phase I, CHMM takes K sets of weak labels from the provided K weak labeling sources. In phase II, in addition to the existing sources, it takes CHMM Aggregated labels Train with Generalized EM Train with KLD loss Phase I Phase IIBERT predictions BERT-NERBERT embeddings Weak labels 1: BERT Input sentence Weak Source 1 ... ... another set of labels from the previously fine-tuned BERT-NER, making the total number of sources K + 1 . For convenience, we use K as the number of weak sources below. Model Architecture Figure 3 shows a sketch of CHMM's architecture. 2 z (1: T ) denotes the discrete hidden states of CHMM with z ( t )  L , representing the underlying true labels to be inferred from multiple weak annotations. ( t )  R |L||L| is the transition matrix, whose element  ( t ) i,j = p ( z ( t ) = j | z ( t  1) = i, e ( t ) ) , i, j  { 1 , . . . , |L|} denotes the probability of moving from label i to label j at time step t . ( t ) k  R |L||L| is the emission matrix of weak source k , each element in which  ( t ) i,j,k = p ( x ( t ) j,k = 1 | z ( t ) = i, e ( t ) ) represents the probability of source k observing label j when the 2 We relax plate notation here to present details. hidden label is i at time step t . For each step, e ( t )  R d emb is the output of a pre-trained BERT with d emb being its embedding dimension. ( t ) and  ( t ) 1: K are calculated by applying a multi-layer perceptron (MLP) to e ( t ) : s ( t )  R |L| 2 = MLP( e ( t ) ) , (1) h ( t )  R |L||L| K = MLP( e ( t ) ) . To achieve the proper probability distributions, we apply the Softmax function along the label axis so that these values are positive and sum up to 1 : ( t ) i, 1: |L| =  ( S ( t ) i, 1: |L| ) ,  ( t ) i, 1: |L| ,k =  ( H ( t ) i, 1: |L| ,k ) , where  ( a ) i = exp ( a i ) (cid:80) j exp ( a j ) . (5) a is an arbitrary vector. The formulae in the following discussion always depend on e (1: T ) , but we will omit the dependency term for simplicity. Model Training According to the generative process of CHMM, the joint distribution of the hidden states and the observed weak labels for one sequence p ( z (0: T ) , x (1: T ) |  ) can be factorized as: p ( z (0: T ) , x (1: T ) |  ) = p ( z (0) ) p ( x (1: T ) | z (1: T ) ) = p ( z (0) ) T (cid:89) t =1 p ( z ( t ) | z ( t  1) ) T (cid:89) t =1 p ( x ( t ) | z ( t ) ) , (6) where  represents all the trainable parameters. HMM is generally trained with an expectation-maximization (EM, also known as Baum-Welch) algorithm. In the expectation step (E-step), we compute the expected complete data log likelihood: Q (  ,  old ) (cid:44) E z [ (cid:96) c (  ) |  old ] . (7)  old is the parameters from the previous training step, E z [  ] is the expectation over variable z , and (cid:96) c (  ) (cid:44) log p ( z (0: T ) , x (1: T ) |  ) is the comptelete data log likelihood. Let  ( t )  R |L| be the observation likelihood where  ( t ) i (cid:44) p ( x ( t ) | z ( t ) = i ) = K (cid:89) k =1 |L| (cid:88) j =1  ( t ) i,j,k x ( t ) j,k . (8) Combining (6)(8) together, we have Q (  ,  old ) = |L| (cid:88) i =1  (0) i log  i + T (cid:88) t =1 |L| (cid:88) i =1 |L| (cid:88) j =1  ( t ) i,j log  ( t ) i,j + T (cid:88) t =1 |L| (cid:88) i =1  ( t ) i log  ( t ) i , (9) where  1 = 1 ,  2: |L| = 0 ; 3  ( t ) i (cid:44) p ( z ( t ) = i | x (1: T ) ) is the smoothed marginal;  ( t ) i,j (cid:44) p ( z ( t  1) = i, z ( t ) = j | x (1: T ) ) is the expected number of transitions. These parameters are computed using the forward-backward algorithm. 4 In the maximization step (M-step), traditional HMM updates parameters  HMM = {  ,  ,  } by optimizing (7) with pseudo-statistics. 5 However, as the transitions and emissions in CHMM are not standalone parameters, we cannot directly optimize CHMM by this method. Instead, we update the model parameters through gradient descent w.r.t.  CHMM using (9) as the objective function:   CHMM = Q (  CHMM ,  oldCHMM )   CHMM . In practice, the calculation is conducted in the logarithm domain to avoid the loss of precision issue that occurs when the floating-point numbers become too small. To solve the label sparsity issue, i.e. , some entities are only observed by a minority of the weak 3 This assumes the initial hidden state is always O . sources, we modify the observations x (1: T ) before training. If one source k observes an entity at time step t : x ( t ) j (cid:54) =1 ,k > 0 , the observation of nonobserving sources at t will be modified to x ( t ) 1 , = (cid:15) ; x ( t ) j (cid:54) =1 , = (1  (cid:15) ) / |L| ,    { 1 , . . . , K }\ k , where (cid:15) is an arbitrary small value. Note that x ( t ) 1 , corresponds to the observed label O . CHMM Initialization Generally, HMM has its transition and emission probabilities initialized with the statistics   and   computed from the observation set. But it is impossible to directly set  ( t ) and  ( t ) in CHMM to these values, as these matrices are the output of the MLPs rather than standalone parameters. To address this issue, we choose to pre-train the MLPs before starting CHMM's training by minimizing the mean squared error (MSE) loss between their outputs and the target statistics: (cid:96) MSE = 1 T (cid:88) t (cid:107)    S ( t ) (cid:107) 2 F + (cid:107)    H ( t ) (cid:107) 2 F , where (cid:107)  (cid:107) F is the Frobenius norm. Right after initialization, MLPs can only output similar probabilities for all time steps:  ( t )    ,  ( t )    ,  t  { 1 , 2 , . . . , T } . But their token-wise prediction divergence will emerge when CHMM has been trained. The initial hidden state z (0) is fixed to O as it has no corresponding token. most probable sequence of hidden labels  z (1: T ) along with the probabilities of all labels y  (1: T ) . z (1: T ) = arg max z (1: T ) p   CHMM ( z (1: T ) | x (1: T ) 1: K , e (1: T ) ) , y  ( t ) i = p   CHMM ( z ( t ) = i | x (1: T ) 1: K , e (1: T ) ) , where   CHMM represents the trained parameters. These results can be calculated by either the Viterbi decoding algorithm (Viterbi, 1967) or directly maximizing the smoothed marginal  (1: T ) . The pre-trained BERT model encodes semantic and structural knowledge, which can be distilled to further refine the denoised labels from CHMM. Specifically, we construct the BERT-NER model by stacking a feed-forward layer and a Softmax layer on top of the original BERT to predict the probabilities of the classes that each token belongs to (Sun et al., 2019). The probability predictions of CHMM, y  (1: T ) , often referred to as soft labels , are chosen to supervise the fine-tuning procedure. Compared with the hard labels  z (1: T ) , soft labels lead to a more stable training process and higher model robustness (Thiel, 2008; Liang et al., 2020). We train BERT-NER by minimizing the Kullback-Leibler divergence (KL divergence) between the soft labels y  and the model output y :   BERT = arg min  BERTD [ y  (1: T ) (cid:107) y (1: T ) ] = arg min  BERTT (cid:88) t =1 |L| (cid:88) i =1 y  ( t ) i log y  ( t ) i y ( t ) i , (11) where  BERT denotes all the trainable parameters in the BERT model. We obtain the refined labels  y (1: T )  RT |L| from the fine-tuned BERT-NER directly through a forward pass. Different from CHMM, we continue BERT-NER's training with parameter weights from the last loop's checkpoint so that the model is initialized closer to the optimum. Correspondingly, phase II trains BERT-NER with a smaller learning rate, fewer epoch iterations, and batch gradient descent instead of the mini-batch version. 6 This strategy speeds up phase II training without sacri-ficing the model performance as y  (1: T ) does not change significantly from loop to loop. We benchmark CHMM-ALT on four datasets against state-of-the-art weakly supervised NER baselines, including both distant learning models and multi-source label aggregation models. We also conduct a series of ablation studies to evaluate the different components in CHMM-ALT's design. Datasets We consider four NER datasets covering the general, technological and biomedical domains: 1) CoNLL 2003 (English subset) (Tjong Kim Sang and De Meulder, 2003) is a general domain dataset containing 22 , 137 sentences manually labelled with 4 entity types. 2) LaptopReview dataset (Pontiki et al., 2014) consists of 3 , 845 sentences with laptop-related entity mentions. 3) NCBI-Disease dataset (Dogan et al., 2014) contains 793 PubMed abstracts annotated with disease 6 Hyper-parameter values are listed in appendix C. Co03 NCBI CDR LR # Instance 22 , 137 793 1 , 500 3 , 845 # Training 14 , 041 593 500 2 , 436 # Development 3 , 250 100 500 609 # Test 3 , 453 100 500 800 Ave# Tokens 14 . mentions. 4) BC5CDR (Li et al., 2016), the dataset accompanies the BioCreative V CDR challenge, consists of 1 , 500 PubMed articles, annotated with chemical disease mentions. Table 1 shows dataset statistics, including the average number of tokens, entities and weak labeling sources. We use the original word tokens in the dataset if provided and use NLTK (Bird and Loper, 2004) otherwise for sentence tokenization. For weak labeling sources, we use the ones from Lison et al. (2020) for CoNLL 2003, and the ones from Safranchik et al. (2020) for LaptopReview, NCBI-Disease and BC5CDR. 7 Baselines We compare our model to the following state-of-the-art baselines: 1) Majority Voting returns the label for a token that has been observed by most of the sources and randomly chooses one if it's a tie; 2) Snorkel (Ratner et al., 2017) treats each token in a sequence as i.i.d. and conducts the label classification without considering its context; 3) SwellShark (Fries et al., 2017) improves Snorkel by predicting all the target entity spans before classifying them using nave Bayes; 4) AutoNER (Shang et al., 2018) augments distant supervision by predicting whether two consecutive tokens should be in the same entity span; 5) BOND (Liang et al., 2020) adopts self-training and high-confidence selection to further boost the distant supervision performance. 6) HMM is the multi-observation generative model used in Lison et al. (2020) that does not have the integrated neural network; 7) Linked HMM (Safranchik et al., 2020) uses linking rules to provide additional inter-token structural information to the HMM model. For the ablation study, we modify CHMM to another type of i.i.d. model by taking away its transition matrices. This model, named CHMM-i.i.d. , 7 Details are presented in appendix B. Models CoNLL 2003 NCBI-Disease BC5CDR LaptopReview Supervised BERT-NER  (cid:92) 90.74 (90.37/91.10) 88.89 (87.05/90.82) 88.81 (87.12/90.57) 81.34 (82.02/80.67) best consensus (cid:92) 89.18 (100.0/80.47) 81.60 (100.0/68.91) 87.58 (100.0/77.89) 77.72 (100.0/63.55) SwellShark (noun-phrase)  -67.10 (64.70/69.70) 84.23 (84.98/83.49) SwellShark (hand-tuned)  -80.80 (81.60/80.10) 84.21 (86.11/82.39) AutoNER  67.00 (75.21/60.40) 75.52 (79.42/71.98) 82.13 (83.23/81.06) 65.44 (72.27/59.79) Snorkel  66.40 (71.40/62.10) 73.41 (71.10/76.00) 82.24 (80.23/84.35) 63.54 (64.09/63.09) Linked HMM  -79.03 (83.46/75.05) 82.96 (82.65/83.28) 69.04 (77.74/62.11) BOND-MV  (cid:92) 65.96 (64.22/67.82) 80.33 (84.77/76.34) 83.18 (82.90/83.49) 67.19 (68.90/65.75) Majority Voting  (cid:92) 58.40 (49.01/72.24) 73.94 (79.76/68.91) 80.73 (83.79/77.88) 67.92 (72.93/63.55) HMM  (cid:92) 68.84 (70.80/66.98) 73.06 (83.88/64.70) 80.57 (88.75/73.76) 66.96 (77.46/58.96) CHMM-i.i.d.  (cid:92) 68.57 (69.67/67.50) 71.69 (83.49/62.87) 79.37 (85.68/73.92) 65.89 (75.70/58.34) CHMM  (cid:92) 70.11 (72.98/67.47) 78.88 ( 93.37 /68.28) 82.39 ( 89.93 /76.02) 73.02 ( 87.23 /62.79) CHMM + BERT-NER  (cid:92) 74.30 (75.02/73.58) 82.87 (89.42/77.22) 84.33 (85.58/83.12) 69.67 (75.48/64.70) CHMM-ALT  (cid:92) 75.54 ( 76.22 / 74.86 ) 85.02 (87.92/ 82.47 ) 85.12 (84.97/ 85.28 ) 76.55 (81.39/ 72.32 ) Table 2: Evaluation results on four datasets. directly predicts the hidden steps from the BERT embeddings, while otherwise identical to CHMM. We also investigate how CHMM-ALT performs with other aggregators other than CHMM. We also introduce two upper bounds from different aspects: 1) a fully supervised BERT-NER model trained with manually labeled data is regarded as a supervised reference; 2) the best possible consensus of the weak sources. The latter assumes an oracle that always selects the correct annotations from these weak supervision sources. According to the definition, its precision is always 100% and its recall is non-decreasing with the in-crease of the number of weak sources. Evaluation Metrics We evaluate the performance of NER models using entity-level precision, recall, and F1 scores. All scores are presented as percentages. The results come from the average of 5 trials with different random seeds. Implementation Details We use BERT pretrained on different domains for different datasets, both for embedding construction and as the component of the supervised BERT-NER model. The original BERT (Devlin et al., 2019) is used for CoNLL 2003 and LaptopReview datasets, bioBERT (Lee et al., 2019) for NCBI-Disease and SciBERT (Belt-agy et al., 2019) for BC5CDR. Instances with lengths exceeding BERT's maximum length limitation ( 512 ) are broken into several shorter segments. The only tunable hyper-parameter in CHMM is the learning rate. But its influence is negligible benefitted from the stability of the generalized EM, the model is guaranteed to converge to a local optimum if the learning rate is small enough. For all the BERT-NER models used in our experiments, the hyper-parameters except the batch size are fixed to the default values (appendix C). To prevent overfitting, we use a two-scale early stopping strategy for model choosing at two scales based on the development set. The micro-scale early stopping chooses the best model parameters for each individual training process of both CHMM and BERT-NER; the macro-scale early stopping selects the best-performing model in phase II iterations, which reports the test results. In our experiments, phase II exits if the macro-scale development score has not increased in 5 loops or the maximum number of loops ( 10 ) is reached. Table 2 presents the model performance from different domains. We find that our alternate-training framework outperforms all weakly supervised baseline models. In addition, CHMM-ALT approaches or even exceeds the best source consensus, which sufficiently proves the effectiveness of the design. For general HMM-based label aggregators such as CHMM, it is impossible to exceed the best consensus since they can only predict an entity observed by at least one source. Based on this fact, CHMM is designed to select the most accurate observations from the weak sources without shrinking their coverage. In comparison, BERT's language 68 70 72 74 76 PIPII1 PII2 PII3 PII4 PII5 PII6 PII7 PII8 PII9 PII10 CoNLL 2003 CHMMBERT-NERStrongest Baseline representation ability enables it to generalize the entity patterns and successfully discovers those entities annotated by none of the sources. Comparing CHMM + BERT to CHMM, we can conclude that BERT basically exchanges recall with precision, and its high-recall predictions can improve the result of CHMM in return. The complementary nature of these two models is why CHMM-ALT improves the overall performance of weakly supervised NER. Looking at Table 2, we notice that CHMM performs the best amongst all generative models including majority voting, HMM and CHMM-i.i.d. The performance of conventional HMM is largely limited by the Markov assumption with the unchanging transition and emission probabilities. The results in the table validate that conditioning the model on BERT embedding alleviates this limitation. However, the transition matrices in HMM are indispensable, implied by CHMM-i.i.d.'s results, as they provide supplemental information about how the underlying true labels should evolve. Performance Evolution Figure 4 reveals the details of the alternate-training process. For less ambiguous tasks including NCBI-Disease, BC5CDR and LaptopReview with fewer entity types, BERT generally has better performance in phase I but gets surpassed in phase II. Interestingly, BERT's performance never exceeds that of CHMM on the LaptopReview dataset. This may be because BERT fails to construct sufficiently representative patterns from the denoised labels for this dataset. For CoNLL 2003, where it is harder for the labeling sources to model the language structures, the strength of a pre-trained language model in pattern recognition becomes more prominent. From the reModels Co03 NCBI CDR Laptop MV  (cid:92) 58.40 73.94 80.73 67.92 MV-ALT  (cid:92) 66.64 80.83 82.78 70.45 HMM  (cid:92) 68.84 73.06 80.57 66.96 HMM-ALT  (cid:92) 74.04 82.99 83.34 72.90 i.i.d.  (cid:92) 68.57 71.69 79.37 65.89 i.i.d.-ALT  (cid:92) 73.84 83.15 83.17 72.61 CHMM  (cid:92) 70.11 78.88 82.39 73.02 CHMM-ALT  (cid:92) 75.54 85.02 85.12 76.55 Table 3: Alternate-training F1 scores with different label aggregators. sults it seems that the performance increment of the denoised labels y  (1: T ) provides marginally extra information to BERT after phase II, as most of the increment comes from the information provided by BERT itself. Even so, keeping phase II is reasonable when we want to get the best out of the weak labeling sources and the pre-trained BERT. BERT-NER Initialization CHMM-ALT initializes BERT-NER's parameters from its previous checkpoint at the beginning of each loop in phase II to reduce training time ( 4.3). If we instead fine-tune BERT-NER from the initial parameters of the pre-trained BERT model for each loop, CHMM-ALT gets 84 . 30 , 84 . 71 , and 76 . 68 F1 scores on NCBI-Disease, BC5CDR, and LaptopReview datasets. These scores are close to the results in Table 2, but the training takes much longer. Consequently, our BERT-NER initialization strategy is a more practical choice overall. Applying Alternate-Training to Other Methods Table 3 shows the alternate-training performance acquired with different label aggregators. The accompanying BERT-NER models are identical to those described in  5.1. The results in the table suggest that the performance improvement obtained by using alternate-training on the label aggregators is stable and generalizable to any other models yet to be proposed. In this work, we present CHMM-ALT, a multi-source weakly supervised approach that does not depend on manually labeled data to learn an accurate NER tagger. It integrates a label aggregator CHMM and a supervised modelBERT-NER together into an alternate-training procedure. CHMM conditions HMM on BERT embeddings to achieve greater flexibility and stronger context-awareness. Fine-tuned with CHMM's prediction, BERT-NER discovers patterns unobserved by the weak sources and complements CHMM. Training these models in turn, CHMM-ALT uses the knowledge encoded in both the weak sources and the pre-trained BERT model to improve the final NER performance. In the future, we will consider imposing more constraints on the transition and emission probabilities, or manipulating them according to sophisticated domain knowledge. This technique could be also extended to other sequence labeling tasks such as semantic role labeling or event extraction. This work was supported by ONR MURI N00014-17-1-2656, NSF III-2008334, Kolon Industries, and research gifts from Google and Amazon. In addition, we would like to thank Yue Yu for his insightful suggestions for this work.
method abstain objective abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective objective method other other other other other other other other other other other other other other other other other other other other other method abstain other method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain method abstain abstain other abstain abstain method abstain abstain abstain method other abstain abstain abstain method abstain abstain abstain abstain other abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method abstain method other abstain method method abstain abstain method abstain abstain method abstain abstain abstain abstain method method abstain method abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain other other