Fine-grained Entity typing (FGET) is the task of assigning a fine-grained type from a hierarchy to entity mentions in the text. As the taxonomy of types evolves continuously, it is desirable for an entity typing system to be able to recognize novel types without additional training. This work proposes a zero-shot entity typing approach that utilizes the type description available from Wikipedia to build a distributed semantic representation of the types. During training, our system learns to align the entity mentions and their corresponding type representations on the known types. At test time, any new type can be incorporated into the system given its Wikipedia descriptions. We evaluate our approach on FIGER, a public benchmark entity tying dataset. Because the existing test set of FIGER covers only a small portion of the fine-grained types, we create a new test set by manually annotating a portion of the noisy training data. Our experiments demonstrate the effectiveness of the proposed method in recognizing novel types that are not present in the training data. Entity Typing assigns a semantic type (e.g., person , location , organization ) to an entity mention in text based on the local context. It is useful for enhancing a variety of Natural Language Process-ing(NLP) tasks such as question answering (Han et al., 2017; Das et al., 2017), relation extraction (Liu et al., 2014; Yaghoobzadeh et al., 2016), and entity linking (Stern et al., 2012). Traditional Named Entity Typing systems consider a small set of coarse types (e.g., person , location , organization ) (Tjong Kim Sang and De Meulder, 2003; Krishnan and Manning, 2006; Chieu and Ng, 2002). Recent studies address larger sets of fine-grained types organized in type hierarchies (e.g., per-son/artist , person/author ) (Ling and Weld, 2012; Corro et al., 2015; Xu and Barbosa, 2018; Murty et al., 2018). Fine-Grained Entity Typing (FGET) is usually approached as a multi-label classification task where an entity mention can be assigned multiple types that usually constitute a path in the hierarchy (Ren et al., 2016). In real-world scenarios, there is a need to deal with ever-growing type taxonomies. New types emerge, and existing types are refined into finer sub-categories. Traditional methods for entity typing assume that the training data contains all possible types, thus require new annotation effort for each new type that emerges. Zero-shot learning (ZSL) , a special kind of transfer learning, allows for new types to be incorporated at the prediction stage without the need for additional annotation and retraining. The main idea behind ZSL is to learn a shared semantic space for representing both the seen and unseen types, which allows the knowledge about how examples link to the seen types to be transferred to unseen types. For fine-grained entity types, we observe that their associated Wikipedia pages often provide a rich description of the types. To capture this, we propose a Description-based Zero-shot Entity Typing (DZET) approach that utilizes the Wikipedia description of each type (e.g., see https://en.wikipedia.org/wiki/ Artist for description of the type person/artist ) to generate a representation of that type. We learn to project the entity-mention representations and the type representations into a shared semantic space, such that the mention is closer to the correct type(s) than the incorrect types. The mid-level type representation derived from the Wikipedia page along with the learned projection function allows the system to recognize new types requiring zero training examples. We investigate different approaches for constructing the type representation based on Wikipedia descriptions. Note that the descriptions can be quite long, often containing many different parts that are useful for recognizing different entity mentions. This motivates us to generate a bag of representations for each type and apply average pooling to aggregate the results. We evaluate the performance of our methods on FIGER, a benchmark dataset for the FNET task, in which types are organized in 2-levels hierarchy. In this work, We focus on testing our method's capability in recognizing unseen fine-grained types ( Level-2 types in this dataset). As the current test set of FIGER contains examples from only a few level-2 types, we created a new test data that covers most of the level-2 types by manually annotating a portion of the noisy training data. Below we summarize our main contributions. We proposed a description-based zero-shot fine-grained entity typing framework that uses Wikipedia descriptions to represent and detect novel types unseen in training. We created a new test set for fine-grained entity typing that provides much better coverage of the level-2 (fine-grained) types compared to the original FIGER test data. We provided experimental evidence of the effectiveness of our approach in comparison with established baselines. Existing work on FGET focuses on performing context-sensitive typing (Gillick et al., 2014; Corro et al., 2015), learning from noisy training data (Abhishek et al., 2017; Ren et al., 2016; Xu and Barbosa, 2018), and exploiting the type hierarchies to improve the learning and inference (Yo-gatama et al., 2015; Murty et al., 2018). More recent studies support even finer granularity (Choi et al., 2018; Murty et al., 2018). However, all the methods above have the limitation that they assume all types are present during training. Zero-Shot Learning has been extensively studied in Computer Vision (CV) (Wang et al., 2019) for tasks such as image classification (Lampert et al., 2014; Zhang and Saligrama, 2015; Socher et al., 2013), object localization (Li et al., 2014, 2017) and image retrieval (Xu et al., 2017; Zhang et al., 2018). A common approach for zero-shot learning in CV is to represent each class (e.g., Zebra) by a set of semantic attributes such as its shape and color. The semantic attributes serve as the intermediate level that connects the visual features with the classes. The model is trained to learn an alignment between the semantic attributes and the visual features where a new class can be recognized using its semantic attributes without the need for any training examples. In contrast, this type of approach tends not to work well for NLP applications as the semantic concepts/classes in NLP are often more complex and cannot be easily described by a set of pre-defined attributes. This explains why the few studies of ZSL for NLP use very different methods to create the transferable intermediate representations. Zero-Shot Learning has been studied for a number of NLP tasks including event extraction (Huang et al., 2018; Lee and Jha, 2018; Srivastava et al., 2018), relation extraction(Liu et al., 2014), Conversational Language Understanding (Lee and Jha, 2018). Specifically, Zero shot entity typing has also been explored, where most of the prior methods adopt the idea of learning a shared semantic space for representing the entities as well as the types, but differ in how they construct the type embeddings. In OTyper (Yuan and Downey, 2018), each type is represented by averaging the embedding of the words constitutes the type label. On the other hand, ProtoLE (Ma et al., 2016) represents each type by a prototype that consists of manually selected entity mentions, where the type embedding is obtained by averaging the prototype mentions' word embeddings.In contrast, our work differs from OTyper and ProtoLE by constructing the type representations based on the Wikipedia descriptions of the types, which not only carry more information about the type but also can be easily adapted to other tasks such as event typing and text classification. Following prior work on fine-grained entity typing, we formulate it as a multi-class multi-label classification problem. Given an entity mention m along with its left textual context c l and right context c r , We learn a classifier that predicts a binary label vector y  { 0 , 1 } | L | , where L denotes the set of all types, which forms a hierarchy  . Here y ( t ) = 1 if the mention m is of type t , and 0 otherwise. In the case of zero-shot FGET, new types can be introduced and added to L during testing. We will begin by introducing our typing function that is used to compute a score between a given mention and type pair, given their corresponding vector representations. We will discuss how to construct the representations in later sections. Formally, the input to this typing function consists of the representation of the mention, denoted by x  R d ; and the representation of a candidate type t , denoted by y t  R  d . It computes a bi-linear score for the ( x, y t ) pair as follows: f ( x, y t , W ) = x TW y t where W  R d   d is a compatibility matrix. Following (Yogatama et al., 2015; Ma et al., 2016), we factorize W as a product of two low-rank matrices to reduce the number of parameters. That is W = ATB , where A  R h  d and B  R h   d (We use h = 20 ). The scoring function f can be rewritten as: f ( x, y t , A, B ) =  ( x, A )   ( y t , B ) = ( Ax ) T By t where  ( x, A ) : x  Ax and  ( y t , B ) : y t  By t serve as the projection functions that map x and y t into a shared semantic space. To obtain the representation for entity mentions, we adopt the same neural approach proposed by Shimaoka et al. (2017). Given an entity mention with its context, we compute a vector v m to present the mention m itself, and another vector v c to represent its left and right contexts c l and c r . v m is computed by simply averaging the embedding of the individual words in m . To compute the context embedding v c , we first encode c l and c r using a bidirectional-LSTM. Let c l 1 , ..., c l s and c r 1 , ..., c r s be the word embedding of the left and the right context respectively, where s is the window size (we use s = 10 ), the output layer of the bi-LSTM is denoted as:  h l 1 ,  h l 1 ...,  h ls ,  h ls and  h r 1 ,  h r 1 ...,  h rs ,  h rs . We then compute a scalar attention for each context word using a 2-level feedforward neural network: e ji = tanh ( W e    h ji  h ji   );  a ji = exp ( W a e ji ) Where W e  R d h  2  d a , W a  R 1  d a , d h is the dimension of LSTM, d a is the attention dimension, j  { l, r } . Next, we normalize a ji s such that they sum up to 1. i.e., a ji =  a ji (cid:80) si =1 ( a li + a ri ) . Finally the context representation is computed as v c = (cid:80) si =1 ( a li (cid:34)  h li  h li (cid:35) + a ri (cid:34)  h ri  h ri (cid:35) ) . The final representation of the entity mention x  R d is a concatenation of v m and v c . Let P t be the Wikipedia page that is used to build a representation for type t . Some types do not have a Wikipedia page with a title the same as the type label. In such cases, we manually look for a Wikipedia page of a similar concept. For example, we represent the type living-thing by the Wikipedia page organism . To get a type representation, We started by the simplest possible method which is averaging the embedding of words in the Wikipedia page ( we call this Avg encoder ). Since some words in the Wikipedia page carry more of the type semantic than the other words we also consider a (tf-idf)-weighted version of the Avg encoder. Learning multiple representations. Wikipedia descriptions are often long and contain multiple parts, where different parts may capture different aspects of the type and relate to different mentions. Moreover, sequence models such as LSTM cannot be applied to such long sequences. This motivates us to consider the approach of constructing a bag of multiple representations for each type based on its Wikipedia description. To obtain a bag of representations for type t , we first use a fixed-length window to incrementally break P t into multiple parts, one paragraph at a time. If a paragraph fits in the current Window, it is added. Otherwise, a new window is initiated. Each window of text r ti is then used to generate one representation. To construct an embedding for r ti , we adopt the same Bidirectional LSTM and attention mechanism that are used to embed the mention context. To compute the score for type t given its multiple representations, we compute the score with each individual representation and average them to produce the final score. This is equivalent to applying average pooling to the multiple representations to obtain a single representation due to the bi-linear typing function. Given the training data, we jointly train the representation and the scoring function by minimizing a ranking score. Let Y ( i ) and Y ( i ) denote the set of correct and incorrect types assigned to the example x ( i ) respectively, we learn to score types in Y ( i ) higher than types in Y ( i ) with a multi-label max-margin ranking objective as follows: (cid:88) y Y (cid:88)  y Y max (0 , 1  f ( x, y, A, B )+ f ( x,  y, A, B )) At testing, both seen and unseen types are mapped to their learned representations, which are then scored for a given input. Given the scores, we conduct a top-down search following the type hierarchy  . Starting from the root we recursively find the type with the highest score among the children. Since we focus on the fine-grained types, we stop the search when a leaf type is reached and predict that the mention is positive for all types along the path leading the to leaf type. Datasets. Our experiments use FIGER, a publicly available fine-grained entity typing benchmark dataset in which types are organized into a 2-level hierarchy. The training data consists of sentences sampled from Wikipedia articles and automatically annotated via distant supervision (Ling and Weld, 2012). The test data consisting of manually annotated sentences sampled from news reports. Setting. To evaluate our capability to recognize fine-grained types in zero-shot setting, we assume all second-level types are unseen during training, i.e., we remove all level-2 types from the train and dev data but keep them in the test data. We observe that the FIGER test set covers only a small number of second-level types. This renders it insufficient for testing under the evaluation setting we adopt. Moreover, the training data is noisy since it is automatically annotated by distant supervision. As a result, we cannot just use part of it for testing. To overcome this limitation, We manually annotated a new test set from the noisy training data. We first divide the train set into clean and noisy as suggested in (Ren et al., 2016). Clean examples are those whose types fall on a single path (not necessarily ending with a leaf) in  . For instance, the mention with labels person , person/author , and person/doctor is considered as noisy example because the labels form two paths. We then manually verify the correctness of up to 20 examples from the clean training data for every level-2 type. These examples are removed from training and added to the test set. We ignore the types with no clean examples. The statistics of the new and original datasets are reported in Table 1. Baselines. We consider two baselines that employ the same neural architecture but use different type representations. The Label embd baseline use the average of the embedding of the words in the type label as the type representation. ProtoLE baseline uses the prototypes-based label embedding learned by Ma et al. (2016), where each type is represented by the set of the most representative entity mentions. The type embedding is the average of all mentions in the corresponding prototype. Evaluation metrics. Following prior works in FGET, we report Accuracy (StrictF 1 ) , loose Macro-averaged F 1 ( F 1 ma ) and loose Micro-averaged F 1 ( F 1 mi ) (Ling and Weld, 2012). The training and hyperparameter tuning details are described in the Appendices. Results and discussions. Table 2 presents the results on FIGER, evaluated on all types (Over-all), the seen types (Level-1) and the unseen types (Level-2) respectively. From the results, we can see that our description based methods have a particularly strong advantage over baselines on level-2 types. This is consistent with our expectation because Wikipedia descriptions tend to be highly informative for fine-grained types, but less so for coarser types. Among the average encoders, we found that weighting the word embedding by the word tf-idf produces better results than treating the words equivalently. As expected, using LSTM based multi-representation adds a noticeable benefit to our system as it produces the best performance among all tested methods, achieving the best performance for level-2 types and outperforming oth-Approach Overall Level-1 Level-2 Acc F 1 ma F 1 mi F 1 ma F 1 mir F 1 ma F 1 mir Label embd 0.2846 0.5510 0.5603 0.8165 0.8163 0.2854 0.2954 ProtoLE 0.2541 0.4982 0.5093 0.7424 0.7422 0.2541 0.2657 DZET + Avg encoder 0.3141 0.5522 0.5614 0.7903 0.7902 0.3141 0.3247 DZET + Weighted Avg encoder 0.3261 0.5500 0.5607 0.7740 0.7738 0.3261 0.3390 DZET + Multi-rep 0.3806 0.5953 0.6045 0.8100 0.8098 0.3806 0.3926 Table 2: Level-1 , Level-2 and overall performance of the models on FIGER dataset. The effect of description quality. Figure 1 analyzes the relationship between the length of the Wikipedia description as one criterion of the description quality and the performance of Multi-rep method. In particular, we group the types based on the length of their Wikipedia descriptions and provide the five-number summary box plot of the F-scores for each group. It can be readily observed that the performance is low when the description of the type's Wikipedia page is too short ( < 1000 words) or too long ( > 4000 words). Short descriptions are less informative and carry less shared semantics with the type's mentions. On the other hand, overly long descriptions could also be confusing as it might share a significant number of common words with the descriptions of other types. A closer look into the results unveils some exceptions. For example, the F-score on the type /education/educational-degree' is 0.7742 even it has a long description (6845 words). The description of this type is indeed very informative and includes a comprehensive list of the educational degrees awarded all around the world. factor that affects the performance of DZET methods. One factor is the performance on the Level-1 types. Since the inference is performed by following the type hierarchy, if an incorrect type is inferred at level-1, there is no hope to get the correct level-2 type. Another factor is the amount of overlapping between the descriptions of the related types. For instance, Multi-rep produces zero F-score on the types/event/protest' and /lo-cation/province' because they share a lot of common words with the types /event/attack' and /lo-cation/county' respectively, which negatively affects the ability of Multi-rep to distinguish between the related types. Both /event/protest' and /location/province' have a description length between 2000 and 3000 words. To mitigate the effect of the contents overlapping between the highly related type, We plan to apply mention-sensitive attention mechanisms for future work to aggregate the scores in Multi-rep instead of max-pooling. In this paper, we propose a novel zero-shot entity typing approach that uses Wikipedia descriptions to construct type embeddings. Our architecture relies on the type embeddings to make predictions for unseen types. Experimental results demonstrate the effectiveness of the proposed methods. We thank Jordan University of Science and Technology for Ph.D. fellowship (to R. O.).
abstain abstain objective abstain abstain method objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain abstain objective abstain objective method method objective objective objective objective method other other other other other other other other other other other other other method method abstain abstain method method abstain abstain method method other method method abstain method method method method abstain abstain abstain abstain abstain method method method method abstain abstain abstain method method abstain abstain abstain method method abstain method method method method method abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain other