Distantly-labeled data can be used to scale up training of statistical models, but it is typically noisy and that noise can vary with the distant labeling technique. In this work, we propose a two-stage procedure for handling this type of data: denoise it with a learned model, then train our final model on clean and denoised distant data with standard supervised training. Our denoising approach consists of two parts. First, a filtering function discards examples from the distantly labeled data that are wholly unusable. Second, a relabeling function repairs noisy labels for the retained examples. Each of these components is a model trained on synthetically-noised examples generated from a small manually-labeled set. We investigate this approach on the ultra-fine entity typing task of Choi et al. (2018). Our baseline model is an extension of their model with pre-trained ELMo representations, which already achieves state-of-the-art performance. Adding distant data that has been denoised with our learned models gives further performance gains over this base model, outperforming models trained on raw distant data or heuristically-denoised distant data. With the rise of data-hungry neural network models, system designers have turned increasingly to unlabeled and weakly-labeled data in order to scale up model training. For information extraction tasks such as relation extraction and entity typing, distant supervision (Mintz et al., 2009) is a powerful approach for adding more data, using a knowledge base (Del Corro et al., 2015; Rabinovich and Klein, 2017) or heuristics (Ratner et al., 2016; Hancock et al., 2018) to automatically label instances. One can treat this data just like any other supervised data, but it is noisy; more effective approaches employ specialized probabilistic models (Riedel et al., 2010; Ratner et al., 2018a), capturing its interaction with other supervision (Wang and Poon, 2018) or breaking down aspects of a task on which it is reliable (Ratner et al., 2018b). However, these approaches often require sophisticated probabilistic inference for training of the final model. Ideally, we want a technique that handles distant data just like supervised data, so we can treat our final model and its training procedure as black boxes. This paper tackles the problem of exploiting weakly-labeled data in a structured setting with a two-stage denoising approach. We can view a distant instance's label as a noisy version of a true underlying label. We therefore learn a model to turn a noisy label into a more accurate label, then apply it to each distant example and add the resulting denoised examples to the supervised training set. Critically, the denoising model can condition on both the example and its noisy label, allowing it to fully leverage the noisy labels, the structure of the label space, and easily learnable correspondences between the instance and the label. Concretely, we implement our approach for the task of fine-grained entity typing, where a single entity may be assigned many labels. We learn two denoising functions: a relabeling function takes an entity mention with a noisy set of types and returns a cleaner set of types, closer to what manually labeled data has. A filtering function discards examples which are deemed too noisy to be useful. These functions are learned by taking manually-labeled training data, synthetically adding noise to it, and learning to denoise, similar to a conditional variant of a denoising autoencoder (Vincent et al., 2008). Our denoising models embed both entities and labels to make their predictions, mirroring the structure of the final entity typing model itself. tity typing scenario and use the same two distant supervision sources as them, based on entity linking and head words. On top of an adapted model from Choi et al. (2018) incorporating ELMo (Pe-ters et al., 2018), navely adding distant data actually hurts performance. However, when our learned denoising model is applied to the data, performance improves, and it improves more than heuristic denoising approaches tailored to this dataset. Our strongest denoising model gives a gain of 3 F 1 absolute over the ELMo baseline, and a 4.4 F 1 improvement over naive incorporation of distant data. This establishes a new state-of-the-art on the test set, outperforming concurrently published work (Xiong et al., 2019) and matching the performance of a BERT model (Devlin et al., 2018) on this task. Finally, we show that denoising helps even when the label set is projected onto the OntoNotes label set (Hovy et al., 2006; Gillick et al., 2014), outperforming the method of Choi et al. (2018) in that setting as well. We consider the task of predicting a structured target y associated with an input x . Suppose we have high-quality labeled data of n (input, target) pairs D = { (cid:0) x (1) , y (1) (cid:1) , . . . , ( x ( n ) , y ( n ) ) } , and noisily labeled data of n (cid:48) (input, target) pairs D (cid:48) = { ( x (1) , y (1) noisy ) , . . . , ( x ( n (cid:48) ) , y ( n (cid:48) ) noisy ) } . For our tasks, D is collected through manual annotation and D (cid:48) is collected by distant supervision. We use two models to denoise data from D (cid:48) : a filtering function f disposes of unusable data (e.g., mislabeled examples) and a relabeling function g transforms the noisy target labels y noisy to look more like true labels. This transformation improves the noisy data so that we can use it to D without introducing damaging amounts of noise. In the second stage, a classification model is trained on the augmented data ( D combined with denoised D (cid:48) ) and predicts y given x in the inference phase. The primary task we address here is the fine-grained entity typing task of Choi et al. (2018). Instances in the corpus are assigned types from a vocabulary of more than 10,000 types, which are divided into three classes: 9 general types, 121 fine-grained types, and 10 , 201 ultra-fine types. This dataset consists of 6K manually annotated examples and approximately 25M distantly-labeled examples. 5M examples are collected using entity linking (EL) to link mentions to Wikipedia and gather types from information on the linked pages. 20M examples (HEAD) are generated by extracting nominal head words from raw text and treating these as singular type labels. Figure 1 shows examples from these datasets which illustrate the challenges in automatic annotation using distant supervision. The manually-annotated example in (a) shows how numerous the gold-standard labeled types are. By contrast, the HEAD example (b) shows that simply treating the head word as the type label, while correct in this case, misses many valid types, including more general types. The EL example (c) is incorrectly annotated as region , whereas the correct coarse type is actually person . This error is characteristic of entity linking-based distant supervision since identifying the correct link is a challenging problem in and of itself (Milne and Witten, 2008): in this case, Gascoyne is also the name of a region in Western Australia. The EL example in (d) has reasonable types; however, human annotators could choose more types (grayed out) to describe the mention more precisely. The average number of types annotated by humans is 5 . 4 per example while the two distant supervision techniques combined yields 1 . 5 types per example on average. In summary, distant supervision can (1) produce Djokovic lost to [ Rafael Nadal ] on Monday, ... player tennis_player A person who participates in or is skilled at some game An athlete who plays tennis + + Rafael Nadal Good Bad  tennis player winner athlete baseball player person Yes Yes Yes No Yes  Filter Relabel Figure 2: Denoising models. completely incorrect types, and (2) systematically miss certain types. To handle the noisy data, we propose to learn a denoising model as shown in Figure 2. This denoising model consists of filtering and relabeling functions to discard and relabel examples, respectively; these rely on a shared mention encoder and type encoder, which we describe in the following sections. The filtering function is a binary classifier that takes these encoded representations and predicts whether the example is good or bad. The relabeling function predicts a new set of labels for the given example. We learn these functions in a supervised fashion. Training data for each is created through synthetic noising processes applied to the manually-labeled data, as described in Sections 3.3 and 3.4. For the entity typing task, each example ( x, y ) takes the form (( s, m ) , t ) , where s is the sentence, m is the mention span, and t is the set of types (either clean or noisy). This encoder is a function  m ( s, m ) which maps a sentence s and mention m to a real-valued vector v m . This allows the filtering and relabeling function to recognize inconsistencies between the given example and the provided types. Note that these inputs s and m are the same as the inputs for the supervised version of this task; we can therefore share an encoder architecture between our denoising model and our final typing model. We use an encoder following Choi et al. (2018) with a few key differences, which are described in Section 4. The second component of our model is a module which produces a vector v t =  t ( t ) . This is an encoder of an unordered bag of types. Our basic type encoder uses trainable vectors as embeddings for each type and combines these with summing. That is, the noisy types t 1 , . . . , t m are embedded into type vectors { t 1 , . . . , t m } . The final embedding of the type set t = (cid:80) j t j . Type Definition Encoder Using trainable type embeddings exposes the denoising model to potential data sparsity issues, as some types appear only a few or zero times in the training data. Therefore, we also assign each type a vector based on its definition in WordNet (Miller, 1995). Even low-frequent types are therefore assigned a plausible embedding. 1 Let w ji denote the i th word of the j th type's most common WordNet definition. Each w ji is embedded using GloVe (Pennington et al., 2014). The resulting word embedding vectors w ji are fed into a bi-LSTM (Hochreiter and Schmidhu-ber, 1997; Graves and Schmidhuber, 2005), and a concatenation of the last hidden states in both directions is used as the definition representation w j . The final representation of the definitions is the sum over these vectors for each type: w = 1 We found this technique to be more effective than using pretrained vectors from GloVe or ELMo. It gave small improvements on an intrinsic evaluation over not incorporating it; results are omitted due to space constraints. k w . Our final v t = [ t ; w ] , the concatenation of the type and definition embedding vectors. The filtering function f is a binary classifier designed to detect examples that are completely mislabeled. Formally, f is a function mapping a labeled example ( s, m, t ) to a binary indicator z of whether this example should be discarded or not. In the forward computation, the feature vectors v m and v t are computed using the mention and type encoders. The model prediction is defined as P ( error ) =  (cid:0) u (cid:62) Highway ([ v m ; v t ]) (cid:1) , where  is a sigmoid function, u is a parameter vector, and Highway (  ) is a 1-layer highway network (Srivas-tava et al., 2015). We can apply f to each distant pair in our distant dataset D (cid:48) and discard any example predicted to be erroneous ( P ( error ) > 0 . 5 ). Training data We do not know a priori which examples in the distant data should be discarded, and labeling these is expensive. We therefore construct synthetic training data D error for f based on the manually labeled data D . For 30% of the examples in D , we replace the gold types for that example with non-overlapping types taken from another example. The intuition for this procedure follows Figure 1: we want to learn to detect examples in the distant data like Gascoyne where heuristics like entity resolution have misfired and given a totally wrong label set. Formally, for each selected example (( s, m ) , t ) , we repeatedly draw another example (( s (cid:48) , m (cid:48) ) , t (cid:48) ) from D until we find t (cid:48) error that does not have any common types with t . We then create a positive training example (( s, m, t (cid:48) error ) , z = 1) . We create a negative training example (( s, m, t ) , z = 0) using the remaining 70% of examples. f is trained on D error using binary cross-entropy loss. The relabeling function g is designed to repair examples that make it through the filter but which still have errors in their type sets, such as missing types as shown in Figure 1b and 1d. g is a function from a labeled example ( s, m, t ) to an improved type set  t for the example. Our model computes feature vectors v m and v t by the same procedure as the filtering function f . The decoder is a linear layer with parameters D  R | V t | ( d m + d t ) . We compute e =  ( D [ v m ; v t ]) , where  is an element-wise sigmoid operation designed to give binary probabilities for each type. Once g is trained, we make a prediction  t for each ( s, m, t )  D (cid:48) and replace t by  t to create the denoised data D (cid:48) denoise = { ( s, m,  t ) , . . . } . For the final prediction, we choose all types t (cid:96) where e (cid:96) > 0 . 5 , requiring at least two types to be present or else we discard the example. Training data We train the relabeling function g on another synthetically-noised dataset D drop generated from the manually-labeled data D . To mimic the type distribution of the distantly-labeled examples, we take each example ( s, m, t ) and randomly drop each type with a fixed rate 0 . 7 independent of other types to produce a new type set t (cid:48) . We perform this process for all examples in D and create a noised training set D drop , where a single training example is (( s, m, t (cid:48) ) , t ) . g is trained on D (cid:48) drop with a binary classification loss function over types used in Choi et al. (2018), described in the next section. In this section, we define the sentence and mention encoder  m , which is use both in the denoising model as well as in the final prediction task. We extend previous attention-based models for this task (Shimaoka et al., 2017; Choi et al., 2018). At a high level, we have an instance encoder  m that returns a vector v m  R d  , then multiply the output of this encoding by a matrix and apply a sigmoid to get a binary prediction for each type as a probability of that type applying. Figure 3 outlines the overall architecture of our typing model. The encoder  m consists of four vectors: a sentence representation s , a word-level mention representation m word , a character-level mention representation m char , and a headword mention vector m head . The first three of these were employed by Choi et al. (2018). We have modified the mention encoder with an additional bi-LSTM to better encode long mentions, and additionally used the headword embedding directly in order to focus on the most critical word. These pieces use pretrained contextualized word embeddings (ELMo) (Peters et al., 2018) as input. using ELMo; let s (cid:48) i  R d ELMo denote the embedding of the i th word. As suggested in Peters et al. (2018), we learn task specific parameters  task  R and s task  R 3 governing these embeddings. We do not fine-tune the parameters of the ELMo LSTMs themselves. Sentence Encoder Following Choi et al. (2018), we concatenate the m th word vector s m in the sentence with a corresponding location embedding (cid:96) m  R d loc . Each word is assigned one of four location tokens, based on whether (1) the word is in the left context, (2) the word is the first word of the mention span, (3) the word is in the mention span (but not first), and (4) the word is in the right context. The input vectors [ s (cid:48) ; (cid:96) ] are fed into a bi-LSTM encoder, with hidden dimension is d hid , followed by a span attention layer (Lee et al., 2017; Choi et al., 2018): s = Attention ( bi-LSTM ([ s (cid:48) ; l ])) , where s is the final representation of the sentence s . Mention Encoder To obtain a mention representation, we use both word and character information. For the word-level representation, the mention's contextualized word vectors m (cid:48) are fed into a bi-LSTM with hidden dimension is d hid . The concatenated hidden states of both directions are summed by a span attention layer to form the word-level mention representation: m word = Attention ( bi-LSTM ( m (cid:48) )) . Second, a character-level representation is computed for the mention. Each character is embedded and then a 1-D convolution (Collobert et al., 2011) is applied over the characters of the mention. This gives a character vector m char . Finally, we take the contextualized word vector of the headword m head as a third component of our representation. This can be seen as a residual connection (He et al., 2016) specific to the mention head word. We find the headwords in the mention spans by parsing those spans in isolation using the spaCy dependency parser (Honnibal and Johnson, 2015). Empirically, we found this to be useful on long spans, when the span attention would often focus on incorrect tokens. The final representation of the input x is a concatenation of the sentence, the word& character-level mention, and the mention headword representations, v = (cid:2) s ; m word ; m char ; m head (cid:3)  R d  . Decoder We treat each label prediction as an independent binary classification problem. Thus, we compute a score for each type in the type vocabulary V t . Similar to the decoder of the relabeling function g , we compute e =  ( E v ) , where E  R | V t | d  and e  R | V t | . For the final prediction, we choose all types t (cid:96) where e (cid:96) > 0 . 5 . If none of e (cid:96) is greater than 0 . 5 , we choose t (cid:96) = arg max e (the single most probable type). Loss Function We use the same loss function as Choi et al. (2018) for training. This loss partitions the labels in general, fine, and ultra-fine classes, and only treats an instance as an example for types of the class in question if it contains a label for that class. More precisely: L = L general 1 general ( t ) + L fine 1 fine ( t ) + L ultra-fine 1 ultra-fine ( t ) , (1) where L ... is a loss function for a specific type class: general, fine-grained, or ultra-fine, and 1 ... ( t ) is an indicator function that is active when one of the types t is in the type class. Each L ... is a sum of binary cross-entropy losses over all types in that category. That is, the typing problem is viewed as independent classification for each type. Note that this loss function already partially repairs the noise in distant examples from missing labels: for example, it means that examples from HEAD do not count as negative examples for general types when these are not present. However, we show in the next section that this is not suffi-cient for denoising. hyper-parameters in our model largely follows Choi et al. (2018) and recommendations for using the pretrained pretrained ELMo-Small model. 2 The word embedding size d ELMo is 1024 . The type embedding size and the type definition embedding size are set to 1024 . For most of other model hyperparameters, we use the same settings as Choi et al. (2018): d loc = 50 , d hid = 100 , d char = 100 . The number of filters in the 1-d convolutional layer is 50 . Dropout is applied with p = 0 . 2 for the pretrained embeddings, and p = 0 . 5 for the mention representations. We limit sentences to 50 words and mention spans to 20 words for computational reasons. The character CNN input is limited to 25 characters; most mentions are short, so this still captures subword information in most cases. The batch size is set to 100 . For all experiments, we use the Adam optimizer (Kingma and Ba, 2014). The initial learning rate is set to 2e-03. We implement all models 3 using PyTorch. To use ELMo, we consult the AllenNLP source code. Ultra-Fine Entity Typing We evaluate our approach on the ultra-fine entity typing dataset from Choi et al. (2018). The 6K manually-annotated English examples are equally split into the training, development, and test examples by the authors of the dataset. We generate synthetically-noised data, D error and D drop , using the 2K training set to train the filtering and relabeling functions, f and h . We randomly select 1M EL and 1M HEAD examples and use them as the noisy data D (cid:48) . Our augmented training data is a combination of the manually-annotated data D and D (cid:48) denoised . OntoNotes In addition, we investigate if denoising leads to better performance on another dataset. We use the English OntoNotes dataset (Gillick et al., 2014), which is a widely used benchmark for fine-grained entity typing systems. The original training, development, and test splits contain 250K, 2K, and 9K examples respectively. Choi et al. (2018) created an augmented training set that has 3.4M examples. We also construct our own augmented training sets with/without denoising using our noisy data D (cid:48) , using the same label mapping from ultra-fine types to OntoNotes types described in Choi et al. (2018). We first compare the performance of our approach to several benchmark systems, then break down the improvements in more detail. We use the model architecture described in Section 4 and train it on the different amounts of data: manually labeled only, naive augmentation (adding in the raw distant data), and denoised augmentation. We compare our model to Choi et al. (2018) as well as to BERT (Devlin et al., 2018), which we fine-tuned for this task. We adapt our task to BERT by forming an input sequence  [CLS] sentence [SEP] mention [SEP]  and assign the segment embedding A to the sentence and B to the mention span. 4 Then, we take the output vector at the position of the [CLS] token (i.e., the first token) as the feature vector v , analogous to the usage for sentence pair classification tasks. The BERT model is fine-tuned on the 2K manually annotated examples. We use the pretrained BERT-Base, uncased model 5 with a step size of 2e-05 and batch size 32. Results Table 1 compares the performance of these systems on the development set. Our model with no augmentation already matches the system of Choi et al. (2018) with augmentation, and incorporating ELMo gives further gains on both precision and recall. On top of this model, adding the distantly-annotated data lowers the performance; the loss function-based approach of (Choi et al., 2018) does not sufficiently mitigate the noise in this data. However, denoising makes the distantly-annotated data useful, improving recall by a substantial margin especially in the general class. A possible reason for this is that the relabeling function tends to add more general types given finer types. BERT performs similarly to ELMo with denoised distant data. As can be seen in the performance breakdown, BERT gains from improvements in recall in the fine class. Table 2 shows the performance of all settings on the test set, with the same trend as the performance on the development set. Our approach outperforms the concurrently-published Xiong et al. (2019); however, that work does not use ELMo. Their improved model could be used for both de-4 We investigated several approaches, including taking the head word piece from the last layer and using that for classification (more closely analogous to what Devlin et al. (2018) did for NER), but found this one to work best. Usage of Pretrained Representations Our model with ELMo trained on denoised data matches the performance of the BERT model. We experimented with incorporating distant data (raw and denoised) in BERT, but the fragility of BERT made it hard to incorporate: training for longer generally caused performance to go down after a while, so the model cannot exploit large external data as effectively. Devlin et al. (2018) prescribe training with a small batch size and very specific step sizes, and we found the model very sensitive to these hyperparameters, with only 2e-05 giving strong results. The ELMo paradigm of incorporating these as features is much more flexible and modular in this setting. Finally, we note that our approach could use BERT for denoising as well, but this did not work better than our current approach. Adapting BERT to leverage distant data effectively is left for future work. We now explicitly compare our denoising approach to several baselines. For each denoising method, we create the denoised EL, HEAD, and EL & HEAD dataset and investigate performance on these datasets. Any denoised dataset is combined with the 2K manually-annotated examples and used to train the final model. functions in a non-learned way. SYNONYMS ANDHYPERNYMS For each type observed in the distant data, we add its synonyms and hypernyms using WordNet (Miller, 1995). This is motivated by the data construction process in Choi et al. (2018). COMMONTYPEPAIRS We use type pair statistics in the manually labeled training data. For each base type that we observe in a distant example, we add any type which is seen more than 90% of the time the base type occurs. For instance, the type art is given at least 90% of the times the film type is present, so we automatically add art whenever film is observed. OVERLAP We train a model on the manually-labeled data only, then run it on the distantly-labeled data. If there is an intersection between the noisy types t and the predicted type  t , we combine them and use as the expanded type  t . Inspired by tri-training (Zhou and Li, 2005), this approach adds obvious types but avoids doing so in cases where the model has likely made an error. Results Table 3 compares the results on the development set. We report the performance on each of the EL & HEAD, EL, and HEAD dataset. On top of the baseline ORIGINAL , adding synonyms and hypernyms by consulting external knowledge does not improve the performance. Expanding labels with the PAIR technique results in small gains over ORIGINAL . OVERLAP is the most ef-EL & HEAD EL HEAD Type Denoising Method P R F1 P R F1 P R F1 RAWDATA 55.2 26.4 35.7 52.3 26.1 34.8 52.8 28.4 36.9 Heuristic Baselines SYNONYMS & HYPERNYMS 43.0 30.0 35.3 47.5 26.3 33.9 44.8 31.7 37.1 PAIR 50.2 29.0 36.8 49.6 27.0 35.0 50.6 31.2 38.6 OVERLAP 50.0 32.3 39.2 49.5 30.8 38.0 50.6 31.4 38.7 Proposed Approach FILTER 53.1 28.2 36.8 51.9 26.5 35.1 51.2 31.2 38.7 RELABEL 52.1 32.2 39.8 50.2 31.4 38.6 50.2 31.8 38.9 FILTER & RELABEL 50.7 33.1 40.1 52.7 30.5 38.7 50.7 32.1 39.3 Choi et al. (2018) 48.1 23.2 31.3 50.3 19.6 28.2 48.4 22.3 30.6 Table 3: Macro-averaged P/R/F1 on the dev set for the entity typing task of Choi et al. (2018) with various types of augmentation added. fective heuristic technique. This simple filtering and expansion heuristic improves recall on EL. FILTER , our model-based example selector, gives similar improvements to PAIR and OVERLAP on the HEAD setting, where filtering noisy data appears to be somewhat important. 6 RELABEL and OVERLAP both improve performance on both EL and HEAD while other methods do poorly on EL. Combining the two model-based denoising techniques, FILTER & RELABEL outperforms all the baselines. We compare our different augmentation schemes for deriving data for the OntoNotes standard as well. Table 4 lists the results on the OntoNotes test set following the adaptation setting of Choi et al. (2018). Even on this dataset, denoising signifi-cantly improves over naive incorporation of distant data, showing that the denoising approach is not just learning quirks of the ultra-fine dataset. Our augmented set is constructed from 2M seed examples while Choi et al. (2018) have a more complex procedure for deriving augmented data from 25M examples. Ours (total size of 2.1M) is on par with their larger data (total size of 3.4M), despite having 40% fewer examples. In this setting, BERT still performs well but not as well as our model with augmented training data. One source of our improvements from data augmentation comes from additional data that is able to be used because some OntoNotes type can be derived. This is due to denoising doing a better job 6 One possible reason for this is identifying stray word senses; film can refer to the physical photosensitive object, among other things. of providing correct general types. In the EL setting, this yields 730k usable examples out of 1M (vs 540K for no denoising), and in HEAD, 640K out of 1M (vs. 73K). To understand what our denoising approach does to the distant data, we analyze the behavior of our filtering and relabeling functions. Table 5 reports the average numbers of types added/deleted by the relabeling function and the ratio of examples discarded by the filtering function. Overall, the relabeling function tends to add more and delete fewer number of types. The HEAD examples have more general types added than the EL examples since the noisy HEAD labels are typically finer. Fine-grained types are added to both EL and HEAD examples less frequently. Ultra-fine examples are frequently added to both datasets, with more added to EL; the noisy EL labels are mostly extracted from Wikipedia defini-General Fine Ultra-Fine Data Add Del Add Del Add Del Filter (%) EL 0.87 0.01 0.36 0.17 2.03 0.12 9.4 HEAD 1.18 0.00 0.51 0.01 1.15 0.16 10.0 Table 5: The average number of types added or deleted by the relabeling function per example. tions, so those labels often do not include ultra-fine types. The filtering function discards similar numbers of examples for the EL and HEAD data: 9 . 4% and 10% respectively. Figure 4 shows examples of the original noisy labels and the denoised labels produced by the relabeling function. In example (a), taken from the EL data, the original labels, { location, city } , are correct, but human annotators might choose more types for the mention span, Minneapolis . The relabeling function retains the original types about the geography and adds ultra-fine types about administrative units such as { township, municipality } . In example (b), from the HEAD data, the original label, { dollar } , is not so expressive by itself since it is a name of a currency. The labeling function adds coarse types, { object, currency } , as well as specific types such as { medium of exchange, monetary unit } . In another EL example (c), the relabeling function tries to add coarse and fine types but struggles to assign multiple diverse ultra-fine types to the mention span Michelangelo , possibly because some of these types rarely cooccur ( painter and poet ). Past work on denoising data for entity typing has used multi-instance multi-label learning (Yaghoobzadeh and Schutze, 2015, 2017; Murty et al., 2018). One view of these approaches is that they delete noisily-introduced labels, but they cannot add them, or filter bad examples. Other work focuses on learning type embeddings (Yogatama et al., 2015; Ren et al., 2016a,b); our approach goes beyond this in treating the label set in a structured way. The label set of Choi et al. (2018) is distinct in not being explicitly hierarchical, making past hierarchical approaches difficult to apply. Denoising techniques for distant supervision have been applied extensively to relation extraction. Here, multi-instance learning and probabilis-... play their home games at Target Center in [ Minneapolis ]. (a) location, city location, place, city, country, area, region, township, town, municipality ... Vittoria was influenced also by [ Michelangelo ] ... (c) architect, sculptor, painter, poet person, artist, writer [ The dollar ] has been rising , pushing commodities lower ... dollar (b) object, currency, money, medium of exchange, dollar, monetary unit Figure 4: Examples of the noisy labels (left) and the denoised labels (right) for mentions (bold). tic graphical modeling approaches have been used (Riedel et al., 2010; Hoffmann et al., 2011; Sur-deanu et al., 2012; Takamatsu et al., 2012) as well as deep models (Lin et al., 2016; Feng et al., 2017; Luo et al., 2017; Lei et al., 2018; Han et al., 2018), though these often focus on incorporating signals from other sources as opposed to manually labeled data. In this work, we investigated the problem of denoising distant data for entity typing tasks. We trained a filtering function that discards examples from the distantly labeled data that are wholly unusable and a relabeling function that repairs noisy labels for the retained examples. When distant data is processed with our best denoising model, our final trained model achieves state-of-the-art performance on an ultra-fine entity typing task. This work was partially supported by NSF Grant IIS-1814522, NSF Grant SHF-1762299, a Bloomberg Data Science Grant, and an equipment grant from NVIDIA. The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing HPC resources used to conduct this research. Results presented in this paper were obtained using the Chameleon testbed supported by the National Science Foundation. Thanks as well to the anonymous reviewers for their thoughtful comments, members of the UT TAUR lab and Pengx-iang Cheng for helpful discussion, and Eunsol Choi for providing the full datasets and useful resources.
abstain objective abstain abstain abstain abstain abstain result method abstain abstain abstain abstain abstain method abstain result abstain method method abstain abstain abstain abstain abstain result abstain abstain result method method method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method method method abstain method method abstain abstain abstain abstain abstain method method method abstain method method abstain method abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method abstain abstain method method abstain other abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain method method method abstain method abstain abstain abstain abstain abstain result abstain abstain result result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other method other other other other other other other other objective method result other other other other