yueyu@gatech.edu le.song@mbzuai.ac.ae chaozhang@gatech.edu Abstract Weakly-supervised learning (WSL) has shown promising results in addressing label scarcity on many NLP tasks, but manually designing a comprehensive, high-quality labeling rule set is tedious and difficult. We study interactive weakly-supervised learningthe problem of iteratively and automatically discovering novel labeling rules from data to improve the WSL model. Our proposed model, named PRBOOST , achieves this goal via iterative prompt-based rule discovery and model boosting. It uses boosting to identify large-error instances and then discovers candidate rules from them by prompting pre-trained LMs with rule templates. The candidate rules are judged by human experts, and the accepted rules are used to generate complementary weak labels and strengthen the current model. Experiments on four tasks show PRBOOST outperforms state-of-the-art WSL baselines up to 7 . 1% , and bridges the gaps with fully supervised mod-els.Our Implementation is available at https: //github.com/rz-zhang/PRBoost . Weakly-supervised learning (WSL) has recently attracted increasing attention to mitigate the label scarcity issue in many NLP tasks. In WSL, the training data are generated by weak labeling rules obtained from sources such as knowledge bases, frequent patterns, or human experts. The weak labeling rules can be matched with unlabeled data to create large-scale weak labels, allowing for training NLP models with much lower annotation cost. WSL has recently achieved promising results in many tasks including text classification (Awasthi et al., 2020; Mekala and Shang, 2020; Meng et al., 2020; Yu et al., 2021b), relation extraction (Zhou et al., 2020), and sequence tagging (Lison et al., 2020; Safranchik et al., 2020; Li et al., 2021b). learning process. First, it is challenging to provide a comprehensive and high-quality set of labeling rules a priori. Labeling rules are often human-written (Ratner et al., 2017; Hancock et al., 2018), but the process of writing labeling rules is tedious and time-consuming even for experts. A few works attempt to automatically discover labeling rules by mining labeled data (Varma and R, 2018), or enumerating predefined types. However, the pre-extracted rules are restricted to frequent patterns or predefined types, which are inadequate for training an accurate model. Second, most existing WSL methods are static and can suffer from the noise in the initial weak supervision (Ratner et al., 2017; Zhou et al., 2020; Yu et al., 2021b; Meng et al., 2020; Zhang et al., 2022). As the labeling rule set remains fixed during model training, the initial errors can be amplified, resulting in an over-fitted end model. Interactive rule discovery has been explored in two recent works (Boecking et al., 2021; Galhotra et al., 2021), which solicits human feedback on candidate rules to refine the rule set. Unfortunately, their rule forms are limited to simple repetitive structures such as n -grams (Boecking et al., 2021), and the huge rule search space makes an enumerating-pruning pipeline not scalable for large datasets (Galhotra et al., 2021). Due to the above reasons, state-of-the-art WSL methods still underperform fully-supervised methods by significant gaps on many NLP tasks. As shown in a recent study (Zhang et al., 2021), the best WSL methods fall behind the best fully-supervised methods in 15 out of 18 NLP benchmarks; and the average performance gap is 18 . 84% in terms of accuracy or F1 score. To bridge the gap between weakly-supervised and fully-supervised approaches, we propose an iterative rule discovery and boosting framework, namely PRBOOST for interactive WSL. Compared to existing works on WSL and active learning, PRBOOST features three key designs: 745 First, we design a rule discovery module that uses rule templates for prompting pre-trained language models (PLMs). By feeding difficult instances and rule templates into PLMs, the module distills knowledge from PLMs via prompting and generates candidate rules that capture key semantics of the input instances. Compared to prior works based on n -grams (Boecking et al., 2021), our prompt-based rule discovery is more expressive and applicable to any tasks that support prompting. Second, we design a boosting-style ensemble strategy to iteratively target difficult instances and adaptively propose new rules. In each iteration, we reweigh data by the boosting error to enforce the rule discovery module to focus on larger-error instances. This avoids enumerating all the possible rules and implementing post-filtering for novel rules, but directly targets rule discovery on large-error instances to provide complementary information to the current model. Third, we strategically solicit human feedback to evaluate the candidate rules. Humans are asked to judge whether a candidate rule should be accepted or abstained. The accepted high-quality rules are then used to generate new weak labels that are fed into boosted model training. As the prompt-generated rules are highly interpretable, the rule evaluation is simply a binary choice task for human experts and thus effortless. Unlike traditional active learning methods that annotate individual instances, such a rule-level annotation is more label-efficient because the annotated rules can match large amounts of instances. We compare our method with supervised, weakly-supervised and interactive learning baselines on four tasks: relation extraction, ontology classification, topic classification, and chemical-protein interaction prediction. The results show: 1) Our method outperforms state-of-the-art weakly-supervised baselines by up to 7 . 1% ; 2) The rule-level annotation helps the model achieve higher model performance compared to the instance-level annotation under the same budget; 3) The machine-discovered and human-evaluated rules are of high quality, which consistently refine the weak labels and the model in each iteration. Our key contributions are: (1) a prompt-based rule discovery framework for interactive WSL, which provides flexible rule representation while capturing subtle semantics in rule generation; (2) an iterative boosting strategy for discovering novel rules from hard instances and strengthening the model by an ensemble of complementary weak models; (3) an interpretable and easy-to-annotate interactive process for rule annotation; (4) comprehensive experiments demonstrating the effectiveness of our framework. Weakly-Supervised Learning WSL has recently attracted much attention in various NLP tasks. Despite their promising performance on various tasks, manually designing the rules can be time-consuming. Moreover, the noise and incompleteness of the initial rules could be propagated in model training (Zhang et al., 2021). A few works attempt to reduce human efforts in manually designing labeling rules by discovering rules from data. For example, Snuba (Varma and R, 2018) generates heuristics based on a small labeled dataset with pre-defined rule types; TALLOR (Li et al., 2021a) and GLaRA (Zhao et al., 2021) study rule expansion for NER problem based on lexical information and then select rules based on a hand-tuned threshold. However, these methods discover rules in a static way and are constrained to task-specific rule types. In contrast, our framework discovers rules iteratively from the entire unlabeled dataset, which can refine the rule set and enlarge its diversity on-the-fly. Interactive Learning Our work is related to active learning (AL) as both involve human annotators in the learning process. However, the key difference is that AL labels instances based on various query policies (Holub et al., 2008; Shen et al., 2017; Zhang et al., 2020; Ein-Dor et al., 2020; Mar-gatina et al., 2021; Yu et al., 2021a), while our method does not annotate individual instances, but uses annotated rules to match unlabeled data. This makes our method more label-efficient in leveraging human feedback for creating large-scale labeled data. To the best of our knowledge, only a few works have studied interactive WSL (Boeck-ing et al., 2021; Galhotra et al., 2021; Choi et al., 2021; Hsieh et al., 2022) as in our problem. However, they either use simple n-gram based rules (Boecking et al., 2021; Hsieh et al., 2022) that fail to capture sentence-level semantics, or suffer from a huge searching space for context-free grammar rules (Galhotra et al., 2021). Unlike these works, our method uses flexible rule representations based on prompts, and also uses boosting for targeted rule 746 discovery to avoid enumerating all possible rules and performing post-filtering for novel rules. Language Model Prompting Our work is also related to prompt-based learning for PLMs, which converts the original task to a cloze-style task and leverages PLMs to fill the missing information (Brown et al., 2020; Liu et al., 2021a). Prompting has been explored in various tasks, including text classification (Hu et al., 2021; Han et al., 2021; Schick and Schtze, 2021a,b), information extraction (Lester et al., 2021; Chen et al., 2021) and text generation (Dou et al., 2021; Li and Liang, 2021). Recent works focus on generating better prompt templates or learning implicit prompt embeddings (Gao et al., 2021; Liu et al., 2021b,c). However, none of these works studied prompting for generating weak labels. Our work is orthogonal to them since we do not aim to optimize prompts for the original task, but uses prompts and PLMs as a knowledge source for rule discovery. Problem Formulation Weakly-supervised learning (WSL) creates weak labels for model training by applying labeling rules over unlabeled instances D u . Given an unlabeled instance x  D u , a labeling rule r (  ) maps x into an extended label space: r ( x )  y  Y  { 0 } . Here Y is the original label set for the task, and 0 is a special label indicating x is unmatchable by r . Given a set R of labeling rules, we can apply each rule in R on unlabeled instances to create a weakly labeled dataset D (cid:48) l . However, the initial weak labels D (cid:48) l can be highly noisy and incomplete, which hinder the performance of WSL. We thus study the problem of interactive WSL: how can we automatically discover more high-quality labeling rules to enhance the performance of WSL? Besides D u and D (cid:48) l , we also assume access to a small set of clean labels D l ( |D l | (cid:28) |D u | ) , and the task is to iteratively find a set of new rules for model improvement. In each iteration t , we assume a fixed rule annotation budget B , i.e. , one can propose at most B candidate rules R t = { r j } B j =1 to human experts for deciding whether each rule should be accepted or not. The accepted rules R + t are then used to create new weakly labeled instances D (cid:48) t . From D (cid:48) t  D (cid:48) l , a model m t : X  Y can be trained to boost the performance of the current WSL model. For example, keyword-based rules are widely used to map certain keywords to their highly correlated labels (Boecking et al., 2021; Meng et al., 2020; Mekala and Shang, 2020; Liang et al., 2020). Regular expression is another common rule format, which matches instances with pre-defined surface patterns (Awasthi et al., 2020; Yu et al., 2021b; Zhou et al., 2020). Logical rules (Hu et al., 2016; Li et al., 2021a) perform logical operations (such as conjunction  and negation  ) over atomic rules and can thus capture higher-order compositional patterns. We adopt a prompt-based rule representation (Section 4.1), which is flexible to encompass any existing rule representations. Our prompt-based rule relies on a rule template  (  ) for the target task, which contains a [MASK] token to be filled by a PLM M along with an unlabeled instance x . From the rule template  , each candidate rule can be automatically derived by r = g ( M , , x ) . Such a prompt-based rule representation is highly flexible and can be applied to any NLP tasks that support prompting (see examples in Table 1). Overview PRBOOST is an iterative method for interactive WSL. In each iteration, it proposes candidate rules from large-error instances, solicits human feedback on candidate rules, generates weak labels, and trains new weak models for ensembling. Figure 1 shows the process in one iteration of PRBOOST , which relies on three key components: 1. Candidate rule generation. This component proposes candidate rules to be evaluated by human annotators. Using the small labeled dataset D l , it measures the weakness of the current model by identifying large-error instances on D l , and proposes rules based on these instances using PLM prompting. 2. Rule annotation and weak label creation. This component collects human feedback to improve the weak supervision quality. It takes as input the candidate rules proposed by the previous component, and asks humans to select the high-quality ones. Then the human-selected rules R t are used to generate weak labels for the unlabeled instances D u in a soft-matching way. Target rule proposal on large-error instances We design a boosting-style (Hastie et al., 2009) strategy for generating prompt-based candidate rules. This strategy iteratively checks feature regimes in which the current model m t is weak, and proposes candidate rules from such regimes. We use the small labeled dataset D l to identify hard instances, i.e., where the model tends to make cumulative mistakes during iterative learning. The discovered rules can complement the current rule set R and refine the weak labels, so the next model m t +1 trained on the refined weakly labeled data can perform better in the weak regimes. We initialize the weights of the instances in D l as w i = 1 / |D l | , i = 1 , 2 ,    , |D l | . During the iterative model learning process, each w i is updated as the model's weighted loss on instance x i  D l . Specifically, in iteration t  { 1 ,    , n } , we weigh the samples by w i  w i  e  t I ( y i (cid:54) = m t ( x i )) , i = 1 , 2 , . . . , |D l | . (1) In Equation 1,  t is the weight of model m t , which will be used for both detecting hard instances and model ensembling (Section 4.3). We compute  t from the model's error rate on D l :  t = log 1  err t err t + log( K  1) , (2) where err t is given by err t = |D l | (cid:88) i =1 w i I ( y i (cid:54) = m t ( x i )) / |D l | (cid:88) i =1 w i . (3) Intuitively, a sample x i receives a larger weight w i (Equation 1) if the model ensemble consistently make mistakes on x i . A large error is often caused by poor coverage (unlabeled instances matched by few or no rules) or dominating noise in the local feature regimes (rule-matched labels are wrong). The weights can thus guide the rule generator to target the topn large-error instances X e = { x e i } ni =1 . By proposing rules from such instances, we aim to discover novel rules that can complement the current rule set and model ensemble most effectively. Prompt-based rule proposal For a wide range of NLP tasks such as relation extraction and text classification, we can leverage prompts to construct informative rule templates, which naturally leads to expressive labeling rules for WSL. Motivated by this, we design a rule proposal module based on PLM prompting . We present concrete examples of our prompt-based rules in Table 1. The input instance comes from the large-error in-748 Input : Microsoft is an American technology corporation founded by Bill Gates. stances identified on the clean dataset D l . For each task, we have a task-specific template to reshape the original input for prompting PLMs. The resulting prompt typically includes the original input as the context and a mask token to be filled by the PLMs. The final rule encompasses multiple atomic parts to capture different views of information. Each rule is accompanied by a ground-truth label of the original input instance, such a label will be assigned to the unlabeled instances matched by this rule. For example, as shown in Table 1, the prompt of the relation extraction task can be " entity [MASK] entity ", which rephrases the original input using relation phrases while keeping the key semantics. Take news topic classification as another example, by filling the masked slot in the prompt, PLMs propose candidate keyword-based rules for topic classification. Different from the rules extracted from surface patterns of the corpus ( e.g. , n -gram rules), such a prompt-based rule proposal can generate words that do not appear in the original inputsthis capability is important to model generalization. Given a large-error instance x e i  X e , we first convert it into a prompt by x p i =  ( x e i ) . Such a prompt consists of the key components of the original input and a [MASK] token. By inheriting the original input, we construct context for the [MASK] token to be predicted by a pre-trained LM M . To complete the rule, we feed each x p i to M to obtain the probability distribution of the [MASK] token over the vocabulary V : p ( MASK =  v | x p i ) = exp ( v  M ( x p i )) (cid:80) v V exp ( v  M ( x p i )) , (4) where M (  ) denotes the output vector of M , v is the embedding of the token in the vocabulary V , and  v is the embedding of the predicted masked token. We collect the topk predictions with highest p ( MASK =  v | x p i ) to form the candidate rules. By filling the rules based on x e i with the prompt predictions, we obtain the candidate rule set in iteration t , denoted as R t = { r j } B j =1 . Interactive rule evaluation As the candidate rules R t can be still noisy, PRBOOST thus presents R t to humans for selecting high-quality rules. Specifi-cally, for each candidate rule r j  R t , we present it along with its prompt template x p j to human experts, then they judge whether the rule r j should be accepted or not. Formally, r j is associated with a label d j  { 1 , 0 } . When a rule is accepted ( d j = 1 ), it will be incorporated into the accepted rule set R + for later weak label generation. Weak Label Generation After human evaluation, the accepted rules R + t are used to match unlabeled instances D u . We design a mixed soft-matching procedure for matching rules with unlabeled instances, which combines embedding-based similarity and prompt-based vocabulary similarity. The two similarities complements each other: the embedding-based similarity captures global semantics, while the prompt-based similarity captures local features in terms of vocabulary overlapping. Given a rule r j  R + t and an unlabeled instance x u  D u , we detail the computations of the two similarities below. First, the embedding similarity is computed as the cosine similarity between the rule and instance embeddings (Zhou et al., 2020): s aj = ( e u  e r j ) / ( (cid:107) e u (cid:107)  (cid:107) e r j (cid:107) ) , (5) where e u is the instance embedding of x u and e r j is the rule embedding of r j , both embeddings are obtained from a PLM encoder. Next, to compute the prompt-based similarity, we feed  ( x u ) into the prompting model (Equation 4) and use the topk candidates of the [MASK] position as the predicted vocabulary for instance x u . s bj = | V u  V r j | /k, (6) where V u is the vocabulary of instance x u and V r j is the vocabulary of rule r j . Note that for the unlabeled instance, we have |V u | = k , while for the rule, we have |V r j |  k because human annotators may abstain some candidate predictions. s j = s aj + (1   ) s bj . (7) The instance x u is matched by the rule r j if s j is higher than the matching threshold  obtained on the development set. When x u is matched by multiple rules that provide conflicting labels, we use the one with the highest matching score to assign the weak label. If  j  1 ,    , k , the matching score s j is lower than  , we abstain from labeling the instance x u . In iteration t , with the new rule-matched data D r , we obtain an enlarged weakly labeled dataset D t = D t  1  D r . We fit a weak model m t on D t by optimizing: min  1 |D t | (cid:88) ( x i ,  y i ) D t (cid:96) CE ( m t ( x i ) ,  y i ) , (8) where  y i is the weak label for instance x i , and (cid:96) CE is the cross entropy loss. While the weakly labeled dataset has been enlarged, there are still unmatched instances in D u . To exploit such unlabeled and unmatched instances, we adopt the self-training technique for weak model training (Lee, 2013). The self-training process can propagate information from the matched weak labels to the unmatched instances to improve the model m t . Following previous models (Xie et al., 2016; Yu et al., 2021b), for each instance x i  D u , we generate a soft pseudo-label (cid:101) y ij from the current model m t : (cid:101) y ij = q 2 ij /f j (cid:80) j (cid:48) Y ( q 2 ij (cid:48) /f j (cid:48) ) , f j = (cid:88) i q ij (9) where q i = m t ( x i ) is a probability vector such that q i  RK , and q ij is the j -th entry, j  1 ,    , K . The above process yields a pseudo-labeled (cid:101) D u . We update m t by optimizing: L c ( m t , (cid:101) y ) = 1 | (cid:101) D u | (cid:88) x  (cid:101) D u DKL ( (cid:101) y (cid:107) m t ( x )) , (10) where DKL ( P (cid:107) Q ) = (cid:80) k p k log( p k /q k ) is the Kullback-Leibler divergence. Finally, we incorporate the self-trained weak model into the ensemble model. The final model is a weighted ensemble of the weak models: f  (  ) = n (cid:88) t  t m t , (11) where a weak model m t with a low error rate err t will be assigned a higher coefficient  t according to Equation 2. 5 Experiments 5.1 Experiment Setup Tasks and Datasets We conduct experiments on four benchmark datasets, including TACRED (Zhang et al., 2017) for relation extraction, DBPedia (Zhang et al., 2015) for ontology classification, ChemProt (Krallinger et al., 2017) for chemical-protein interaction classification and AG News (Zhang et al., 2015) for news topic classification. For the initial weak supervision sources, we use the labeling rules provided by existing works: Zhou et al. (2020) for TACRED, Meng et al. (2020) for DBPedia, and Zhang et al. (2021) for Chemprot and AG News. The statistics of the four datasets are shown in table 5. For the development set, we do not directly use the full development set as suggested by the recent works (Gao et al., 2021; Perez et al., 2021). This prevents the model from taking the advantage of the massive number of labeled data in the development set. Instead, we create a real label-scarce scenario and keep the number of sample in validation set D v the same as the limited clean labeled set D l , namely |D v | = |D l | . Baselines We include three groups of baselines: Fully Supervised Baseline : PLM : We use the pre-trained language model RoBERTa-base (Liu et al., 2019) as the backbone and fine-tune it with the full clean labeled data except for ChemProt. On ChemProt, we choose BioBERT (Lee et al., 2020) as the backbone for all the baselines and our model to better adapt to this domain-specific task. The performance of fully supervised methods serves as an upper bound for weakly-supervised methods. Weakly Supervised Baselines : (1) Snorkel (Rat-ner et al., 2017) is a classic WSL model. It aggregates different labeling functions with probabilistic models, then fed the aggregated labels to PLM for the target task. (2) LOTClass (Meng et al., 2020) is a recent model for weakly-supervised text classification. It uses label names to probe PLMs to generate weak labels, and performs self-training using the weak labels for classification. (3) CO-750 Method (Metrics) TACRED (F1) DBpedia (Acc.) ChemProt (Acc.) AG News (Acc.) Supervised Baselines PLM w. SINE (Yu et al., 2021b) is a state-of-the-art method on fine-tuning PLMs with weak supervision. It adopts self-training and contrastive learning to fine-tune LMs with weakly-labeled data. Interactive Learning Baselines : (1) Entropy based AL (Holub et al., 2008) is a simple-yet-effective method for AL which acquires samples with the highest predictive entropy. (2) CAL (Mar-gatina et al., 2021) is the most recent method for active learning. It selects samples has the most diverge predictions from their neighbors for annotation. (3) IWS (Boecking et al., 2021) is an interactive WSL model. It firstly generates n-gram terms as candidate rules, then selects quality rules by learning from humans' feedback. Note that IWS is designed for binary classification, which makes it hard to adapt to classification with multiple labels. Evaluation Protocol To propose rules on large-error instances, we assume access to a dataset D l with a limited number of clean labeled data. For our method, such a clean dataset is only used for identifying large-error instances. For fair comparison, for the WSL baselines, we further fine-tune them using the same clean data and compare with such fine-tuned results. Specifically, we use 5% clean data for TACRED and ChemProt, 0 . 5% for AG News and 0 . 1% for DBPedia. We then implement a 10-iteration rule proposal and weak model training. In each iteration, we identify the top10 large-error instances and propose 100 candidate rules in total ( i.e. , 10 candidate rules per instance). Each rule is annotated by three humans, and the annotated rule labels are majority-voted for later weak label generation. Following the common practice (Zhang et al., 2017, 2021), we use F1 score for TACRED and accuracy for other datasets. Table 2 shows the performance of PRBOOST and the baselines on the four datasets. The results show that PRBOOST outperforms the weakly supervised baselines on all the four datasets. When the weakly supervised baselines are not fine-tuned on D l , PRBOOST outperforms the strongest WSL baseline by 8 . 4% , 7 . 2% , 7 . 3% , 2 . 4% on the four benchmarks. Even when the WSL models are further fine-tuned using clean labeled data, PRBOOST still outperform them by 2 . 4% on average. Compared against supervised baselines, PRBOOST is significantly better than the fine-tuned model on TA-751 2 4 6 8 10 Iterations 82 83 84 85 86 87 88 89 M o d e l A cc u r a c y PRBoost IWS CAL Entropy Figure 3: Results of interactive methods on AG News CRED, ChemProt and AG News when the training data is limited. For the model fine-tuned with 100% training data, we narrow the gap to fully supervised learning, compared to other WS approaches. Comparing the performance gains across datasets, the performance gap between PRBOOST and the baselines is the largest on TACRED, which is the most challenging task among the four with 41 different relation types. ChemProt is the smallest dataset with only 5400 training data, so the gain is larger when the WSL methods are fine-tuned with clean labels. The performance gaps among different methods are small on DBPedia, especially after they are fine-tuned using clean labeled data. DBpedia, being a relatively simple dataset, using only 0 . 1% clean data for fine-tuning RoBERTa already achieves 98% accuracy, and the other WSL methods after fine-tuning perform similarly. It is worth noting that PRBOOST performs strongly across all the tasks because we can easily design a task-specific prompt template to adapt to each task. In contrast, some WSL baselines are difficult to apply to certain tasks. For example, LOTClass achieves strong performance for DBpedia and AGNews as its weak sources are tailored for text classification. However, it is hard to apply it to relation extraction tasks. Similarly, IWS performs well on binary classification problems using n-gram based rules, but the method is only designed for binary classification, making it unsuitable for complex multi-class tasks. In this set of experiments, we benchmark model performance and annotation cost against interactive learning baselines (detailed in Appendix D): IWS, CAL, and Entropy-based AL. As shown in Figure 3, PRBOOST outperforms IWS that also features rule-level annotation by 1 . 2% with very close annotation cost. Our method outperforms the best interactive baseline CAL by 1 . 1% in terms of accuracy, while using about 0 . 6  annotation cost. While annotating model-proposed rule or instances, 2 4 6 8 10 Iterations 50 60 70 80 90 100 R u l e P e r f o r m a n c e Rule Coverage Rule Accuracy 86.0 86.5 87.0 87.5 88.0 88.5 89.0 89.5 90.0 M o d e l A cc u r a c y OursRoBERTa w. we asked all the three annotators to time their annotation. On average, it takes each annotator less than 3 seconds to annotate one rule, while it takes nearly 10 seconds to annotate one instance. Rule-level annotation is much more efficient than instance-level annotation because 1) we show the prompt rather than the original instance to humans, which is shorter and easier to read; 2) upon scanning the prompt, the annotators can swiftly select qualified rules as they only differ at the [MASK] position. This shows that rule-level annotation is an efficient and suitable paradigm for interactive WSL. .71 .77 .73 .66 .65 .71 .75 .79 .60 .68 .71 Table 3: Annotation agreement measured by the Fleiss-Kappa  on AG News. P measures annotation agreement over all categories;  P e computes the quadratic sum of the proportion of assignments to each category. For the annotation agreement, we compute Fleiss' kappa  (Fleiss, 1971) to evaluate the agreement among multiple human annotators. This statistic assesses the reliability of agreement among multiple annotators. = 1 indicates complete agreement over all the annotators, and no agreement results in   0 . As shown in Table 3, we obtained an average  = 0 . 71 , which means the annotators achieve substantial agreement. For each iteration, the  ranges between [0 . 60 , 0 . 79] indicating the stability of the annotation agreement. In this set of experiments, we evaluate the quality of the rules discovered by PRBOOST . Figure 2 visualizes the discovered rules on AG News dataset. We observe that 1) the rules can rectify some mis-classified data, and 2) the rules can complement each other. For the first observation, we can take Figure 2(a) and Figure 2(b) for example. In iteration 0 where new rules have not been proposed, it is obvious that some green data points and purple data points are mixed into the orange cluster. After the first-round rule proposal, PRBOOST has already rectified parts of wrong predictions via rule-matching. This is because our rule proposal is targeted on the large-error instances, such adaptively discovered rules can capture the model's weakness more accurately compared to the simply enumerated rules. For the second observation, we found that more mis-classified data points get matched by the newly discovered rules as the iteration increases. It demonstrates PRBOOST can gradually enlarge the effective rule set by adding complementary rules, which avoids proposing repetitive rules that can not improve the rule coverage. Figure 4 shows the changes in rule accuracy, rule coverage, and model performance in the iterative learning process on AG News. As shown, the model's accuracy increases steadily during learning, which is improved from 86 . 7% to 88 . 9% after 10 iterations. This improvement arises from two key aspects of PRBOOST . First, the enlarged rule set continuously augments weakly labeled data, which provides more supervision for the weak model training. Second, the model ensemble approach refines the previous large errors step by step, resulting in increasing ensemble performance. Regarding the rule coverage and accuracy, we observe the coverage of the rule set is improved from 56 . 4% to 77 . 8% , and rule accuracy from 83 . 1% to 85 . 6% . Such improvements show that PRBOOST can adaptively propose novel rules to complement the previous rule set, which can match more instances that were previously unmatchable. Note that the increased rule converge has not compromised rule accuracy, but rather improved it. The reason is two-fold: (1) the human-in-the-loop evaluation can select high-quality rules for generating new weak labels; (2) for the instances with wrong initial weak labels, PRBOOST can discover more rules for the same instances and correct the weak labels through majority voting. We study the effectiveness of various components in PRBOOST and show the ablation study results in Figure 5. We have the following findings: First, the boosting-based iterative rule discovery strategy is effective. For the "w/o ensemble" setting, we fix the annotation budget B but discover candidate rules from large-error samples in one iteration. The results show the superiority of the iterative strategy in PRBOOST , which brings 1 . 2% 2 4 6 8 10 Iterations 85.5 86.0 86.5 87.0 87.5 88.0 88.5 89.0 P e r f o r m a n c e Supervised Initial WS PRBoost w/o self-training w/o rule w/o ensemble Figure 5: Ablation study on AG News. performance gain. PRBOOST iteratively identifies the current model's weaknesses and proposes rules to strengthen itself, therefore it adaptively discovers more effective rules than static rule discovery. Second, ensembling alone without new rule discovery is not as effective. For the "w/o rule" variant, we do not propose new rules, but ensemble multiple self-trained weak classifiers instead. The final performance drops significantly under this setting by 1 . 5% . It demonstrates the newly proposed rules provide complementary weak supervision to the model. Although simply ensembling multiple weak classifiers also helps WSL, it is not as effective as training multiple complementary weak models as in PRBOOST . Third, self-training benefits learning from new weak labels. For the "w/o self-training" setting, we do not use the self-training technique when learning each weak classifier. The performance deteriorates by 0 . 6% . This is because part of the data are still unmatched after we propose new rules, and self-training leverages the unlabeled data to help the model generalize better. We proposed PRBOOST to iteratively discover prompt-based rules for interactive weakly-supervised learning. Through a boosting-style ensemble strategy, it iteratively evaluates model weakness to identify large-error instances for new rule proposal. From such large-error instances, its prompt-based rule discovery module leads to expressive rules that can largely improve rule coverage while being easy to annotate. The discovered rules complement the current rule set and refine the WSL model continuously. Our experiments on four benchmarks demonstrate that PRBOOST can largely improve WSL and narrow the gaps between WSL models and fully-supervised models. This work was supported by ONR MURI N00014-17-1-2656, NSF IIS-2008334, IIS-2106961, and research awards from Google, Amazon, Facebook, and Kolon Inc.
abstain abstain abstain abstain abstain objective objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain method objective method abstain method abstain abstain abstain abstain method result abstain other other other other other other other objective abstain method method method other objective method other other other objective abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other method abstain abstain method abstain abstain abstain method abstain abstain other method abstain abstain abstain abstain method abstain abstain method abstain method abstain method abstain method method method method abstain method abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain objective abstain abstain abstain objective other