Semi-supervised bootstrapping techniques for relationship extraction from text iteratively expand a set of initial seed instances. Due to the lack of labeled data, a key challenge in bootstrapping is semantic drift: if a false positive instance is added during an iteration, then all following iterations are contaminated. We introduce BREX, a new bootstrapping method that protects against such contamination by highly effective confidence assessment. This is achieved by using entity and template seeds jointly (as opposed to just one as in previous work), by expanding entities and templates in parallel and in a mutually constraining fashion in each iteration and by introducing higher-quality similarity measures for templates. Experimental results show that BREX achieves an F 1 that is 0.13 (0.87 vs. 0.74) better than the state of the art for four relationships. Traditional semi-supervised bootstrapping relation extractors (REs) such as BREDS (Batista et al., 2015), SnowBall (Agichtein and Gravano, 2000) and DIPRE (Brin, 1998) require an initial set of seed entity pairs for the target binary relation. They find occurrences of positive seed entity pairs in the corpus, which are converted into extraction patterns, i.e., extractors , where we define an extractor as a cluster of instances generated from the corpus. The initial seed entity pair set is expanded with the relationship entity pairs newly extracted by the extractors from the text iteratively. The augmented set is then used to extract new relationships until a stopping criterion is met. Due to lack of sufficient labeled data, rule-based systems dominate commercial use (Chiti-cariu et al., 2013). Rules are typically defined by creating patterns around the entities (entity extraction) or entity pairs (relation extraction). Recently, supervised machine learning, especially deep learning techniques (Gupta et al., 2015; Nguyen and Grishman, 2015; Vu et al., 2016a,b; Gupta et al., 2016), have shown promising results in entity and relation extraction; however, they need sufficient hand-labeled data to train models, which can be costly and time consuming for web-scale extractions. Bootstrapping machine-learned rules can make extractions easier on large corpora. Thus, open information extraction systems (Carl-son et al., 2010; Fader et al., 2011; Mausam et al., 2012; Mesquita et al., 2013; Angeli et al., 2015) have recently been popular for domain specific or independent pattern learning. Hearst (1992) used hand written rules to generate more rules to extract hypernym-hyponym pairs, without distributional similarity. For entity extraction, Riloff (1996) used seed entities to generate extractors with heuristic rules and scored them by counting positive extractions. Prior work (Lin et al., 2003; Gupta et al., 2014) investigated different extractor scoring measures. Gupta and Manning (2014) improved scores by introducing expected number of negative entities. Brin (1998) developed the bootstrapping relation extraction system DIPRE that generates extractors by clustering contexts based on string matching. SnowBall (Agichtein and Gravano, 2000) is inspired by DIPRE but computes a TF-IDF representation of each context. BREDS (Batista et al., 2015) uses word embeddings (Mikolov et al., 2013) to bootstrap relationships. Related work investigated adapting extractor scoring measures in bootstrapping entity extraction with either entities or templates (Table 1) as seeds (Table 2). The state-of-the-art relation extractors bootstrap with only seed entity pairs and suffer due to a surplus of unknown extractions and the lack of labeled data, leading to low confidence extractors. This in turn leads to to low confidence in the system output. Prior RE sys-26 BREE Bootstrapping Relation Extractor with Entity pair BRET Bootstrapping Relation Extractor with Template BREJ Bootstrapping Relation Extractor in Joint learning type a named entity type, e.g., person typed entity a typed entity, e.g., (cid:160) Obama, person  entity pair a pair of two typed entities template a triple of vectors ( ~v (cid:1) 1 , ~v 0 , ~v 1 ) and an entity pair instance entity pair and template (types must be the same)  instance set extracted from corpus i a member of  , i.e., an instance x p i q the entity pair of instance i x p i q the template of instance i G p a set of positive seed entity pairs G n a set of negative seed entity pairs G p a set of positive seed templates G n a set of negative seed templates G (cid:160) G p ,G n , G p , G n  k it number of iterations  cat cluster of instances ( extractor ) cat category of extractor   NNHC Non-Noisy-High-Confidence extractor (True Positive)  NNLC Non-Noisy-Low-Confidence extractor (True Negative)  NHC Noisy-High-Confidence extractor (False Positive)  NLC Noisy-Low-Confidence extractor (False Negative) Table 1: Notation and definition of key terms tems do not focus on improving the extractors' scores. In addition, SnowBall and BREDS used a weighting scheme to incorporate the importance of contexts around entities and compute a similarity score that introduces additional parameters and does not generalize well. Contributions. (1) We propose a Joint Bootstrapping Machine 1 (JBM), an alternative to the entity-pair-centered bootstrapping for relation extraction that can take advantage of both entity-pair and template-centered methods to jointly learn extractors consisting of instances due to the occurrences of both entity pair and template seeds. It scales up the number of positive extractions for non-noisy extractors and boosts their confidence scores. We focus on improving the scores for non-noisy-low-confidence extractors, resulting in higher recall . The relation extractors bootstrapped with entity pair, template and joint seeds are named as BREE , BRET and BREJ (Table 1), respectively. (2) Prior work on embedding-based context comparison has assumed that relations have consistent syntactic expression and has mainly addressed synonymy by using embeddings (e.g.,acquired  bought). In reality, there is large variation in the syntax of how relations are expressed, e.g., MSFT to acquire NOK for $8B 1 github.com/pgcool/Joint-Bootstrapping-Machines vs. MSFT earnings hurt by NOK acquisition. We introduce cross-context similarities that compare all parts of the context (e.g., to acquire and acquisition) and show that these perform better (in terms of recall) than measures assuming consistent syntactic expression of relations. (3) Experimental results demonstrate a 13% gain in F 1 score on average for four relationships and suggest eliminating four parameters, compared to the state-of-the-art method. The motivation and benefits of the proposed JBM for relation extraction is discussed in depth in section 2.3. The method is applicable for both entity and relation extraction tasks. However, in context of relation extraction , we call it BREJ . We first introduce the notation and terms (Table 1). Given a relationship like  x acquires y , the task is to extract pairs of entities from a corpus for which the relationship is true. We assume that the arguments of the relationship are typed, e.g., x and y are organizations. We run a named entity tagger in preprocessing, so that the types of all candidate entities are given. The objects the bootstrapping algorithm generally handles are therefore typed entities (an entity associated with a type). For a particular sentence in a corpus that states that the relationship (e.g., acquires) holds between x and y , a template consists of three vectors that represent the context of x and y . ~v (cid:1) 1 represents the context before x , ~v 0 the context between x and y and ~v 1 the context after y . These vectors are simply sums of the embeddings of the corresponding words. A template is typed, i.e., in addition to the three vectors it specifies the types of the two entities. An instance joins an entity pair and a template. The types of entity pair and template must be the same. The first step of bootstrapping is to extract a set of instances from the input corpus. We refer to this set as  . We will use i and j to refer to instances. x p i q is the entity pair of instance i and x p i q is the template of instance i . A required input to our algorithm are sets of positive and negative seeds for either entity pairs ( G p and G n ) or templates ( G p and G n ) or both. We define G to be a tuple of all four seed sets. We run our bootstrapping algorithm for k it iterations where k it is a parameter. A key notion is the similarity between two instances. We will experiment with different similarity measures. The baseline is (Batista et al., 2015)'s measure given in Figure 4, first line: the similarity of two instances is given as a weighted sum of the dot products of their before contexts ( ~v (cid:1) 1 ), their between contexts ( ~v 0 ) and their after contexts ( ~v 1 ) where the weights w p are parameters. We give this definition for instances, but it also applies to templates since only the context vectors of an instance are used, not the entities. The similarity between an instance i and a cluster  of instances is defined as the maximum similarity of i with any member of the cluster; see Figure 2, right, Eq. 5. Again, there is a straightforward extension to a cluster of templates: see Figure 2, right, Eq. 6. The extractors  can be categorized as follows:  NNHC (cid:16) t  P  |  (cid:222)(cid:209) R loomoon non (cid:1) noisy ^ cnf p , G q   cnf u (1)  NNLC (cid:16) t  P  |  (cid:222)(cid:209) R ^ cnf p , G q (cid:160)  cnf u (2)  NHC (cid:16) t  P  |  (cid:127)(cid:222)(cid:209) R loomoon noisy ^ cnf p , G q   cnf u (3)  NLC (cid:16) t  P  |  (cid:127)(cid:222)(cid:209) R ^ cnf p , G q (cid:160)  cnf u (4) where R is the relation to be bootstrapped. The  cat is a member of  cat . For instance, a  NNLC is called as a non-noisy-low-confidence extractor if it represents the target relation (i.e.,  (cid:222)(cid:209) R ), however with the confidence below a certain threshold (  cnf ). Extractors of types  NNHC and  NLC are desirable, those of types  NHC and  NNLC undesirable within bootstrapping. To describe BREX (Figure 1) in its most general form, we use the term item to refer to an entity pair, a template or both. The input to BREX (Figure 2, left, line 01) is a set  of instances extracted from a corpus and G seed , a structure consisting of one set of positive and one set of negative seed items. G yield (line 02) collects the items that BREX extracts in several iterations. In each of k it iterations (line 03), BREX first initializes the cache G cache (line 04); this cache collects the items that are extracted in this iteration. The design of the algorithm balances elements that ensure high recall with elements that ensure high precision. High recall is achieved by starting with the seeds and making three hops that consecutively consider order-1, order-2 and order-3 neighbors ... ... of the seeds. On line 05, we make the first hop: all instances that are similar to a seed are collected where similarity is defined differently for different BREX configurations (see below). The collected instances are then clustered, similar to work on bootstrapping by Agichtein and Gravano (2000) and Batista et al. (2015). On line 06, we make the second hop: all instances that are within  sim of a hop-1 instance are added; each such instance is only added to one cluster, the closest one; see definition of  : Figure 2, Eq. 8. On line 07, we make the third hop: we include all instances that are within  sim of a hop-2 instance; see definition of  : Figure 2, Eq. 7. In summary, every instance that can be reached by three hops from a seed is being considered at this point. A cluster of hop-2 instances is named as extractor . High precision is achieved by imposing, on line 08, a stringent check on each instance before its information is added to the cache. The core function of this check is given in Figure 2, Eq. 9. This definition is a soft version of the following hard max, which is easier to explain: cnf p i,  , G q (cid:19) max t  P  | i P  p  qu cnf p i, , G q We are looking for a cluster  in  that licenses the extraction of i with high confidence. cnf p i, , G q (Figure 2, Eq. 10), the confidence of a single cluster (i.e., extractor)  for an instance, is defined as the product of the overall reliability of  (which is independent of i ) and the similarity of i to  , the second factor in Eq. 10, i.e., sim p i,  q . This factor sim p i,  q prevents an extraction by a cluster whose members are all distant from the instance  even if the cluster itself is highly reliable. The first factor in Eq. 10, i.e., cnf p , G q , assesses the reliability of a cluster  : we compute the ratio N (cid:0) p , G n q N (cid:0) p , G p q , i.e., the ratio between the number of instances in  that match a negative and positive gold seed, respectively; see Figure 3, line (i). If this ratio is close to zero, then likely false positive extractions are few compared to likely true positive extractions. For the simple version of the algorithm (for which we set w n (cid:16) 1 , w u (cid:16) 0 ), this results in cnf p , G q being close to 1 and the reliability measure it not discounted. On the other hand, if N (cid:0) p , G n q N (cid:0) p , G p q is larger, meaning that the relative number of likely false positive extractions is high, then cnf p , G q shrinks towards 0, resulting in progressive discounting of cnf p , G q and leading to non-noisy-low-confidence extractor, particularly for a reliable  . Due to lack of labeled data, the scoring mechanism cannot distinguish between noisy and non-noisy extractors. Therefore, an extractor is judged by its ability to extract more positive and less negative extractions. Note that we carefully designed this precision component to give good assessments while at the same time making maximum use of the available seeds. The reliability statistics are computed on  , i.e., on hop-2 instances (not on hop-3 instances). The ratio N (cid:0) p , G n q N (cid:0) p , G p q is computed on instances that directly match a gold seed  this is the most reliable information we have available. After all instances have been checked (line 08) and (if they passed muster) added to the cache (line 09), the inner loop ends and the cache is merged into the yield (line 10). Then a new loop (lines 0310) of hop-1, hop-2 and hop-3 extensions and cluster reliability tests starts. Thus, the algorithm consists of k it iterations. There is a tradeoff here between  sim and k it . We will give two extreme examples, assuming that we want to extract a fixed number of m instances where m is given. We can achieve this goal either by setting k it =1 and choosing a small  sim , which will result in very large hops. Or we can achieve this goal by setting  sim to a large value and running the algorithm for a larger number of k it . The flexibility that the two hyperparameters k it and  sim afford is important for good performance. The differences and advantages of BREJ over BREE and BRET are: (1) Disjunctive Matching of Instances: The first difference is realized in how the three algorithms match instances with seeds (line 05 in Figure 3). BREE checks whether the entity pair of an instance is one of the entity pair seeds, BRET checks whether the template of an instance is one of the template seeds and BREJ checks whether the disjunction of the two is true. The disjunction facilitates a higher hit rate in matching instances with seeds. The introduction of a few handcrafted templates along with seed entity pairs allows BREJ to leverage discriminative patterns and learn similar ones via distributional semantics. In Figure 1, the joint approach results in hybrid extractors  that contain instances due to seed occurrences  of both entity pairs and templates. (2) Hybrid Augmentation of Seeds: On line 09 in Figure 3, we see that the bootstrapping step is defined in a straightforward fashion: the entity pair of an instance is added for BREE, the template for BRET and both for BREJ. Figure 1 demonstrates I1: 's purchase of I2: 's acquisition of I1: 's purchase of ( BREE ) I1: 's purchase of I2: 's acquisition of ( BRET ) ( BREJ ) Seed Entity Pair: = {<Google, DoubleClick>} Seed Templates: = {[X] 's acquisition of [Y]} Matched Instances: I1: < Google > 's purchase of < DoubleClick > is intriguing. (3) Scaling Up Positives in Extractors: As discussed in section 2.2, a good measure of the quality of an extractor is crucial and N (cid:0) , the number of instances in an extractor  that match a seed, is an important component of that. For BREE and BRET, the definition follows directly from the fact that these are entity-pair and template-centered instantiations of BREX, respectively. However, the disjunctive matching of instances for an extractor with entity pair and template seeds in BREJ (Figure 3 line (i) ) boosts the likelihood of find-ing positive instances. In Figure 5, we demonstrate computing the count of positive instances 30 Relationship Seed Entity Pairs Seed Templates acquired { Adidas;Reebok } , { Google;DoubleClick } , { Widnes;Warrington } , { Hewlett-Packard;Compaq } { [X] acquire [Y] } , { [X] acquisition [Y] } , { [X] buy [Y] } , { [X] takeover [Y] } , { [X] merger with [Y] } founder-of { CNN;Ted Turner } , { Facebook;Mark Zuckerberg } , { Microsoft;Paul Allen } , { Amazon;Jeff Bezos } , { [X] founded by [Y] } , { [X] co-founder [Y] } , { [X] started by [Y] } , { [X] founder of [Y] } , { [X] owner of [Y] } headquartered { Nokia;Espoo } , { Pfizer;New York } , { United Nations;New York } , { NATO;Brussels } , { [X] based in [Y] } , { [X] headquarters in [Y] } , { [X] head office in [Y] } , { [X] main office building in [Y] } , { [X] campus branch in [Y] } affiliation { Google;Marissa Mayer } , { Xerox;Ursula Burns } , { Microsoft;Steve Ballmer } , { Microsoft;Bill Gates } , { [X] CEO [Y] } , { [X] resign from [Y] } , { [X] founded by [Y] } , { [X] worked for [Y] } , { [X] chairman director [Y] } Table 2: Seed Entity Pairs and Templates for each relation. N (cid:0) p , G q for an extractor  within the three systems. Observe that an instance i in  can scale its N (cid:0) p , G q by a factor of maximum 2 in BREJ if i is matched in both entity pair and template seeds. The reliability cnf p , G q (Eq. 11) of an extractor  is based on the ratio N (cid:0) p , G n q N (cid:0) p , G p q , therefore suggesting that the scaling boosts its confidence. In Figure 6, we demonstrate with an example how the joint bootstrapping scales up the positive instances for a non-noisy extractor  , resulting in  NNHC for BREJ compared to  NNLC in BREE. Due to unlabeled data, the instances not matching in seeds are considered either to be ig-nored/unknown N 0 or negatives in the confidence measure (Eq. 11). The former leads to high confidences for noisy extractors by assigning high scores, the latter to low confidences for non-noisy extractors by penalizing them. For a simple version of the algorithm in the illustration, we consider them as negatives and set w n (cid:16) 1 . Figure 6 shows the three extractors (  ) generated and their confidence scores in BREE, BRET and BREJ. Observe that the scaling up of positives in BREJ due to BRET extractions (without w n ) discounts cnf p , G q relatively lower than BREE. The discounting results in  NNHC in BREJ and  NNLC in BREE. The discounting in BREJ is adapted for non-noisy extractors facilitated by BRET in generating mostly non-noisy extractors due to stringent checks (Figure 3, line (i) and 05). Intuitively, the intermixing of non-noisy extractors (i.e., hybrid ) promotes the scaling and boosts recall. The before ( ~v (cid:1) 1 ) and after ( ~v 1 ) contexts around the entities are highly sparse due to large variation in the syntax of how relations are expressed. SnowBall, DIPRE and BREE assumed that the between ( ~v 0 ) context mostly defines the syntactic expression for a relation and used weighted mechanism on the three contextual similarities in ORG-ORG ORG-PER ORG-LOC count 58,500 75,600 95,900 Table 3: Count of entity-type pairs in corpus Parameter Description/ Search Optimal | v (cid:1) 1 | maximum number of tokens in before context 2 | v 0 | maximum number of tokens in between context 6 | v 1 | maximum number of tokens in after context 2  sim similarity threshold [0.6, 0.7, 0.8] 0.7  cnf instance confidence thresholds [0.6, 0.7, 0.8] 0.7 w n weights to negative extractions [0.0, 0.5, 1.0, 2.0] 0.5 w u weights to unknown extractions [0.0001, 0.00001] 0.0001 k it number of bootstrapping epochs 3 dim emb dimension of embedding vector, V 300 PMIPMI threshold in evaluation 0.5 Entity Pairs Ordered Pairs ( OP ) or Bisets ( BS ) OP Table 4: Hyperparameters in BREE, BRET and BREJ pairs, sim match (Figure 4). They assigned higher weights to the similarity in between ( p (cid:16) 0 ) contexts, that resulted in lower recall. We introduce attentive ( max ) similarity across all contexts (for example, ~v (cid:1) 1 p i q ~v 0 p j q ) to automatically capture the large variation in the syntax of how relations are expressed, without using any weights. We investigate asymmetric (Eq 13) and symmetric (Eq 14 and 15) similarity measures, and name them as cross-context attentive (sim cc ) similarity. We re-run BREE (Batista et al., 2015) for baseline with a set of 5.5 million news articles from AFP and APW (Parker et al., 2011). We use processed dataset of 1.2 million sentences (released by BREE) containing at least two entities linked to FreebaseEasy (Bast et al., 2014). We extract four relationships: acquired (ORG-ORG), founder-of (ORG-PER), headquartered (ORG-LOC) and affiliation (ORG-PER) for Organization (ORG), Person (PER) and Location (LOC) entity types. We bootstrap relations in BREE, BRET and BREJ, each with 4 similarity measures using seed entity 31 Relationships # out P R F 1 # out P R F 1 # out P R F 1 # out P R F 1 BREE baseline : BREE+sim match config 2 : BREE+sim asymcc config 3 : BREE+sim sym 1 cc config 4 : BREE+sim sym 2 cc acquired 2687 0.88 0.48 0.62 5771 0.88 0.66 0.76 3471 0.88 0.55 0.68 3279 0.88 0.53 0.66 founder-of 628 0.98 0.70 0.82 9553 0.86 0.95 0.89 1532 0.94 0.84 0.89 1182 0.95 0.81 0.87 headquartered 16786 0.62 0.80 0.69 21299 0.66 0.85 0.74 17301 0.70 0.83 0.76 9842 0.72 0.74 0.73 affiliation 20948 0.99 0.73 0.84 27424 0.97 0.78 0.87 36797 0.95 0.82 0.88 28416 0.97 0.78 0.87 avg 10262 0.86 0.68 0.74 16011 0.84 0.81 0.82 14475 0.87 0.76 0.80 10680 0.88 0.72 0.78 BRET config 5 : BRET+sim match config 6 : BRET+sim asymcc config 7 : BRET+sim sym 1 cc config 8 : BRET+sim sym 2 cc acquired 4206 0.99 0.62 0.76 15666 0.90 0.85 0.87 18273 0.87 0.86 0.87 14319 0.92 0.84 0.87 founder-of 920 0.97 0.77 0.86 43554 0.81 0.98 0.89 41978 0.81 0.99 0.89 46453 0.81 0.99 0.89 headquartered 3065 0.98 0.55 0.72 39267 0.68 0.92 0.78 36374 0.71 0.91 0.80 56815 0.69 0.94 0.80 affiliation 20726 0.99 0.73 0.85 28822 0.99 0.79 0.88 44946 0.96 0.85 0.90 33938 0.97 0.81 0.89 avg 7229 0.98 0.67 0.80 31827 0.85 0.89 0.86 35393 0.84 0.90 0.86 37881 0.85 0.90 0.86 BREJ config 9 : BREJ+sim match config 10 : BREJ+sim asymcc config 11 : BREJ+sim sym 1 cc config 12 : BREJ+sim sym 2 cc acquired 20186 0.82 0.87 0.84 35553 0.80 0.92 0.86 22975 0.86 0.89 0.87 22808 0.85 0.90 0.88 founder-of 45005 0.81 0.99 0.89 57710 0.81 1.00 0.90 50237 0.81 0.99 0.89 45374 0.82 0.99 0.90 headquartered 47010 0.64 0.93 0.76 66563 0.68 0.96 0.80 60495 0.68 0.94 0.79 57853 0.68 0.94 0.79 affiliation 40959 0.96 0.84 0.89 57301 0.94 0.88 0.91 55811 0.94 0.87 0.91 51638 0.94 0.87 0.90 avg 38290 0.81 0.91 0.85 54282 0.81 0.94 0.87 47380 0.82 0.92 0.87 44418 0.82 0.93 0.87 Table 5: Precision ( P ), Recall ( R ) and F 1 compared to the state-of-the-art ( baseline ). pairs and templates (Table 2). See Tables 3, 4 and 5 for the count of candidates, hyperparameters and different configurations, respectively. Our evaluation is based on Bronzi et al. (2012)'s framework to estimate precision and recall of large-scale RE systems using FreebaseEasy (Bast et al., 2014). Also following Bronzi et al. (2012), we use Pointwise Mutual Information (PMI) (Tur-ney, 2001) to evaluate our system automatically, in addition to relying on an external knowledge base. We consider only extracted relationship instances with confidence scores cnf p i,  , G q equal or above 0.5. We follow the same approach as BREE (Batista et al., 2015) to detect the correct order of entities in a relational triple, where we try to identify the presence of passive voice using part-of-speech (POS) tags and considering any form of the verb to be, followed by a verb in the past tense or past participle, and ending in the word by'. We use GloVe (Pennington et al., 2014) embeddings. Table 5 shows the experimental results in the three systems for the different relationships with ordered entity pairs and similarity measures (sim match , sim cc ). Observe that BRET (config 5 ) is precision-oriented while BREJ (config 9 ) recall-oriented when compared to BREE (baseline). We see the number of output instances # out are also higher in BREJ, therefore the higher recall. The BREJ system in the different similarity configurak it # out P R F 1 0.6 1 691 0.99 0.21 0.35 2 11288 0.85 0.79 0.81 0.7 1 610 1.0 0.19 0.32 2 7948 0.93 0.75 0.83 0.8 1 522 1.0 0.17 0.29 2 2969 0.90 0.51 0.65 Table 6: Iterations ( k it ) Vs Scores with thresholds (  ) for relation acquired in BREJ. On an average for the four relations, BREJ in configurations config 9 and config 10 results in F 1 that is 0 . 11 (0.85 vs 0.74) and 0 . 13 (0.87 vs 0.74) better than the baseline BREE. We discover that sim cc improves # out and recall over sim match correspondingly in all three systems. Observe that sim cc performs better with BRET than BREE due to non-noisy extractors in BRET. The results suggest an alternative to the weighting scheme in sim match and therefore, the state-of-the-art (sim cc ) performance with the 3 parameters ( w (cid:1) 1 , w 0 and w 1 ) ignored in bootstrap-32 acquired founder-of headquartered affiliation BRE X E T J E T J E T J E T J # hit 71 682 743 135 956 1042 715 3447 4023 603 14888 15052 Table 8: Disjunctive matching of Instances. Observe that sim asymcc gives higher recall than the two symmetric similarity measures. Table 6 shows the performance of BREJ in different iterations trained with different similarity  sim and confidence  cnf thresholds. Table 7 shows a comparative analysis of the three systems, where we consider and evaluate the extracted relationship instances at different confidence scores. As discussed in section 2.3, BREJ facilitates disjunctive matching of instances (line 05 Figure 3) with seed entity pairs and templates. Table 8 shows # hit in the three systems, where the higher values of # hit in BREJ conform to the desired property. Observe that some instances in BREJ are found to be matched in both the seed types. We analyze the extractors  generated in BREE, BRET and BREJ for the 4 relations to demonstrate the impact of joint bootstrapping. Table 9 shows the attributes of  . We manually annotate the extractors as noisy and non-noisy . We compute ANNLC and the lower values in BREJ compared to BREE suggest fewer non-noisy extractors with lower confidence in BREJ due to the scaled confi-Relationships # out P R F 1 BREE acquired 387 0.99 0.13 0.23 founder-of 28 0.96 0.09 0.17 headquartered 672 0.95 0.21 0.34 affiliation 17516 0.99 0.68 0.80 avg 4651 0.97 0.28 0.39 BRET acquired 4031 1.00 0.61 0.76 founder-of 920 0.97 0.77 0.86 headquartered 3522 0.98 0.59 0.73 affiliation 22062 0.99 0.74 0.85 avg 7634 0.99 0.68 0.80 BREJ acquired 12278 0.87 0.81 0.84 founder-of 23727 0.80 0.99 0.89 headquartered 38737 0.61 0.91 0.73 affiliation 33203 0.98 0.81 0.89 avg 26986 0.82 0.88 0.84 Table 10: BREX+sim match :Scores when w n ignored dence scores. ANNE (higher), ANNLC (lower), AP (higher) and AN (lower) collectively indicate that BRET mostly generates NNHC extractors. AP and AN indicate an average of N (cid:0) p , G l q (line (i) Figure 3) for positive and negative seeds, respectively for  P  in the three systems. Observe the impact of scaling positive extractions ( AP ) in BREJ that shrink N (cid:0) p , G n q N (cid:0) p , G p q i.e., ANP . It facilitates  NNLC to boost its confidence, i.e.,  NNHC in BREJ suggested by AES that results in higher # out and recall (Table 5, BREJ). As discussed, Table 5 shows the performance of BREE, BRET and BREJ with the parameter w n (cid:16) 0 . 5 in computing extractors' confidence cnf p , G q (Eq. 11). In other words, config 9 (Ta-ble 5) is combination of both weighted negative and scaled positive extractions. However, we also investigate ignoring w n p(cid:16) 1 . 0 q in order to demonstrate the capability of BREJ with only scaling positives and without weighting negatives. In Table 10, observe that BREJ outperformed both BREE and BRET for all the relationships due to higher # out and recall. In addition, BREJ scores are comparable to config 9 (Table 5) suggesting that the scaling in BREJ is capable enough to remove the parameter w n . However, the combination of both weighting negatives and scaling positives results in the state-of-the-art performance. Table 11 lists some of the non-noisy extractors (simplified) learned in different configurations to illustrate boosting extractor confidence cnf p , G q . Since, an extractor  is a cluster of instances, therefore to simplify, we show one in-33 config 1 : BREE+sim match cnf p , G q config 5 : BRET+sim match cnf p , G q config 9 : BREJ+sim match cnf p , G q config 10 : BREJ+sim asymcc cnf p , G q acquired [X]acquired[Y] 0.98 [X]acquired[Y] 1.00 [X]acquired[Y] 1.00 acquiredby[X],[Y] : 0.93 [X]takeoverof[Y] 0.89 [X]takeoverof[Y] 1.00 [X]takeoverof[Y] 0.98 takeoverof[X]wouldboost[Y]'searnings : 0.90 [X]'splannedacquisitionof[Y] 0.87 [X]'splannedacquisitionof[Y] 1.00 [X]'splannedacquisitionof[Y] 0.98 acquisitionof[X]by[Y] : 0.95 [X]acquiring[Y] 0.75 [X]acquiring[Y] 1.00 [X]acquiring[Y] 0.95 [X]acquiring[Y] 0.95 [X]hasownedpartof[Y] 0.67 [X]hasownedpartof[Y] 1.00 [X]hasownedpartof[Y] 0.88 ownedby[X]'sparent[Y] 0.90 [X]tookcontrolof[Y] 0.49 [X]'sownershipof[Y] 1.00 [X]tookcontrolof[Y] 0.91 [X]takescontrolof[Y] 1.00 [X]'sacquisitionof[Y] 0.35 [X]'sacquisitionof[Y] 1.00 [X]'sacquisitionof[Y] 0.95 acquisitionof[X]wouldreduce[Y]'sshare : 0.90 [X]'smergerwith[Y] 0.35 [X]'smergerwith[Y] 1.00 [X]'smergerwith[Y] 0.94 [X]-[Y]mergerbetween : 0.84 [X]'sbidfor[Y] 0.35 [X]'sbidfor[Y] 1.00 [X]'sbidfor[Y] 0.97 partof[X]which[Y]acquired : 0.83 founder-of [X]founder[Y] 0.68 [X]founder[Y] 1.00 [X]founder[Y] 0.99 founderof[X],[Y] : 0.97 [X]CEOandfounder[Y] 0.15 [X]CEOandfounder[Y] 1.00 [X]CEOandfounder[Y] 0.99 co-founderof[X]'smillennialcenter,[Y] : 0.94 [X]'sco-founder[Y] 0.09 [X]owner[Y] 1.00 [X]owner[Y] 1.00 ownedby[X]cofounder[Y] 0.95 [X]cofounder[Y] 1.00 [X]cofounder[Y] 1.00 Gatesco-founded[X]withschoolfriend[Y] : 0.99 [X]startedby[Y] 1.00 [X]startedby[Y] 1.00 whoco-founded[X]with[Y] : 0.95 [X]wasfoundedby[Y] 1.00 [X]wasfoundedby[Y] 0.99 toco-found[X]withpartner[Y] : 0.68 [X]begunby[Y] 1.00 [X]begunby[Y] 1.00 [X]wasstartedby[Y],cofounder 0.98 [X]hasestablished[Y] 1.00 [X]hasestablished[Y] 0.99 setup[X]withchildhoodfriend[Y] : 0.96 [X]chiefexecutiveandfounder[Y] 1.00 [X]co-founderandbillionaire[Y] (cid:6) 0.99 [X]co-founderandbillionaire[Y] 0.97 headquartered [X]headquartersin[Y] 0.95 [X]headquartersin[Y] 1.00 [X]headquartersin[Y] 0.98 [X]headquartersin[Y] 0.98 [X]relocateditsheadquartersfrom[Y] 0.94 [X]relocateditsheadquartersfrom[Y] 1.00 [X]relocateditsheadquartersfrom[Y] 0.98 basedat[X]'ssuburban[Y]headquarters : 0.98 [X]headofficein[Y] 0.84 [X]headofficein[Y] 1.00 [X]headofficein[Y] 0.87 headof[X]'soperationsin[Y] : 0.65 [X]basedin[Y] 0.75 [X]basedin[Y] 1.00 [X]basedin[Y] 0.98 branchof[X]companybasedin[Y] 0.98 [X]headquartersbuildingin[Y] 0.67 [X]headquartersbuildingin[Y] 1.00 [X]headquartersbuildingin[Y] 0.94 [X]maincampusin[Y] 0.99 [X]headquartersindowntown[Y] 0.64 [X]headquartersindowntown[Y] 1.00 [X]headquartersindowntown[Y] 0.94 [X]headquartersindowntown[Y] 0.96 [X]branchofficesin[Y] 0.54 [X]branchofficesin[Y] 1.00 [X]branchofficesin[Y] 0.98 [X]'s[Y]headquartersrepresented : 0.98 [X]'scorporatecampusin[Y] 0.51 [X]'scorporatecampusin[Y] 1.00 [X]'scorporatecampusin[Y] 0.99 [X]maincampusin[Y] 0.99 [X]'scorporateofficein[Y] 0.51 [X]'scorporateofficein[Y] 1.00 [X]'scorporateofficein[Y] 0.89 [X],[Y]'scorporate : 0.94 affiliation [X]chiefexecutive[Y] 0.92 [X]chiefexecutive[Y] 1.00 [X]chiefexecutive[Y] 0.97 [X]chiefexecutive[Y]resignedmonday 0.94 [X]secretary[Y] 0.88 [X]secretary[Y] 1.00 [X]secretary[Y] 0.94 workedwith[X]manager[Y] 0.85 [X]president[Y] 0.87 [X]president[Y] 1.00 [X]president[Y] 0.96 [X]votedtoretain[Y]asCEO : 0.98 [X]leader[Y] 0.72 [X]leader[Y] 1.00 [X]leader[Y] 0.85 headof[X],[Y] : 0.99 [X]partyleader[Y] 0.67 [X]partyleader[Y] 1.00 [X]partyleader[Y] 0.87 workingwith[X],[Y]suggested : 1.00 [X]hasappointed[Y] 0.63 [X]executiveeditor[Y] 1.00 [X]hasappointed[Y] 0.81 [X]president[Y]wasfired 0.90 [X]player[Y] 0.38 [X]player[Y] 1.00 [X]player[Y] 0.89 [X]'s[Y]wasfired : 0.43 [X]'ssecretary-general[Y] 0.36 [X]'ssecretary-general[Y] 1.00 [X]'ssecretary-general[Y] 0.93 Chairmanof[X],[Y] : 0.88 [X]hired[Y] 0.21 [X]director[Y] 1.00 [X]hired[Y] 0.56 [X]hired[Y]asmanager : 0.85 Table 11: Subset of the non-noisy extractors (simplified) with their confidence scores cnf p , G q learned in different configurations for each relation. stance (mostly populated) from every  . Each cell in Table 11 represents either a simplified representation of  or its confidence. We demonstrate how the confidence score of a non-noisy extractor in BREE (config 1 ) is increased in BREJ (config 9 and config 10 ). For instance, for the relation acquired , an extractor { [X] acquiring [Y] } is generated by BREE, BRET and BREJ; however, its confidence is boosted from 0 . 75 in BREE (config 1 ) to 0 . 95 in BREJ (config 9 ). Observe that BRET generates high confidence extractors. We also show extractors (marked by : ) learned by BREJ with sim cc (config 10 ) but not by config 1 , config 5 and config 9 . In Table 5, we use ordered pairs of typed entities. Additionally, we also investigate using entity sets and observe improved recall due to higher # out in both BREE and BREJ, comparing correspondingly Table 12 and 5 ( baseline and config 9 ). of both entity-pair-centered and template-centered approaches. We have demonstrated that the joint approach scales up positive instances that boosts the confidence of NNLC extractors and improves recall. The experiments showed that the cross-context similarity measures improved recall and suggest removing in total four parameters. We thank our colleagues Bernt Andrassy, Mark Buckley, Stefan Langer, Ulli Waltinger and Us-ama Yaseen, and anonymous reviewers for their review comments. This research was supported by Bundeswirtschaftsministerium ( bmwi.de ), grant 01MD15010A (Smart Data Web) at Siemens AGCT Machine Intelligence, Munich Germany.
abstain abstain objective abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain objective abstain result abstain abstain abstain result abstain abstain abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method method method abstain method abstain method abstain abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain method method method method method method abstain abstain method method method method method abstain abstain result abstain abstain abstain abstain result abstain other abstain abstain result abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain method method abstain objective abstain other other