We propose a novel method for hierarchical entity classification that embraces ontological structure at both training and during prediction. At training, our novel multi-level learning-to-rank loss compares positive types against negative siblings according to the type tree. During prediction, we define a coarse-to-fine decoder that restricts viable candidates at each level of the ontology based on already predicted parent type(s). We achieve state-of-the-art across multiple datasets, particularly with respect to strict accuracy. 1 1 Introduction Entity typing is the assignment of a semantic label to a span of text, where that span is usually a mention of some entity in the real world. Named entity recognition (NER) is a canonical information extraction task, commonly considered a form of entity typing that assigns spans to one of a handful of types, such as PER , ORG , GPE , and so on. Fine-grained entity typing (FET) seeks to classify spans into types according to more diverse, semantically richer ontologies (Ling and Weld, 2012; Yosef et al., 2012; Gillick et al., 2014; Del Corro et al., 2015; Choi et al., 2018), and has begun to be used in downstream models for entity linking (Gupta et al., 2017; Raiman and Raiman, 2018). Consider the example in Figure 1 from the FET dataset, FIGER (Ling and Weld, 2012). The mention of interest, Hollywood Hills , will be typed with the single label LOC in traditional NER, but may be typed with a set of types { /location , /geography , /geography/mountain } under a fine-grained typing scheme. In these finer-grained typing schemes, types usually form a hierarchy: there are a set of coarse types that lies on 1 Code can be found at https://github.com/ ctongfei/hierarchical-typing . the top levelthese are similar to traditional NER types, e.g. /person ; additionally, there are finer types that are subtypes of these top-level types, e.g. /person/artist or /person/doctor . Most prior work concerning fine-grained entity typing has approached the problem as a multi-label classification problem: given an entity mention together with its context, the classifier seeks to output a set of types, where each type is a node in the hierarchy. Approaches to FET include handcrafted sparse features to various neural architectures (Ren et al., 2016a; Shimaoka et al., 2017; Lin and Ji, 2019, inter alia , see section 2). Perhaps owing to the historical transition from flat NER types, there has been relatively little work in FET that exploits ontological tree structure , where type labels satisfy the hierarchical property : a subtype is valid only if its parent supertype is also valid. We propose a novel method that takes the explicit ontology structure into account, by a multi-level learning to rank approach that ranks the candidate types conditioned on the given entity mention. Intuitively, coarser types are easier whereas finer types are harder to classify: we capture this intuition by allowing distinct margins at each level of the ranking model. FET is usually studied as allowing for sentence-level context in making predictions, notably starting with Ling and Weld (2012) and Gillick et al. (2014), where they created the commonly used FIGER and OntoNotes datasets for FET. While researchers have considered the benefits of document-level (Zhang et al., 2018), and corpus-level (Yaghoobzadeh and Schutze, 2015) context, here we focus on the sentence-level variant for best contrast to prior work. Progress in FET has focused primarily on:  Better mention representations: Starting from sparse hand-crafted binary features (Ling and Weld, 2012; Gillick et al., 2014), the community has moved to distributed representations (Yogatama et al., 2015), to pre-trained word embeddings with LSTMs (Ren et al., 2016a,b; Shimaoka et al., 2016; Abhishek et al., 2017; Shimaoka et al., 2017) or CNNs (Murty et al., 2018), with mention-to-context attention (Zhang et al., 2018), then to employing pre-trained language models like ELMo (Peters et al., 2018) to generate ever better representations (Lin and Ji, 2019). Our approach builds upon these developments and uses state-of-the-art mention encoders. as multi-label classification , without using information in the hierarchical structure, but there are a few exceptions. Ren et al. (2016a) proposed an adaptive margin for learning-to-rank so that similar types have a smaller margin; Xu and Barbosa (2018) proposed hierarchical loss normalization that penalizes output that violates the hierarchical property; and Murty et al. (2018) proposed to learn a subtyping relation to constrain the type embeddings in the type space. In contrast to these approaches, our coarse-to-fine decoding approach strictly guarantees that the output does not violate the hierarchical property, leading to better performance. HYENA (Yosef et al., 2012) applied ranking to sibling types in a type hierarchy, but the number of predicted positive types are trained separately with a meta-model, hence does not support neural end-to-end training. Researchers have proposed alternative FET formulations whose types are not formed in a type hierarchy, in particular Ultra-fine entity typing (Choi et al., 2018; Xiong et al., 2019; Onoe and Durrett, 2019), with a very large set of types derived from phrases mined from a corpus. FET in KB (Jin et al., 2019) labels mentions to types in a knowledge base with multiple relations, forming a type graph. Dai et al. (2019) augments the task with entity linking to KBs. We denote a mention as a tuple x = ( w , l , r ) , where w = ( w 1 ,    , w n ) is the sentential context and the span [ l : r ] marks a mention of interest in sentence w . That is, the mention of interest is ( w l ,    , w r ) . Given x , a hierarchical entity typing model outputs a set of types Y in the type ontology Y , i.e. Y  Y . Type hierarchies take the form of a forest, where each tree is rooted by a top-level supertype (e.g. /person , /location , etc.). We add a dummy parent node ENTITY =  / , the supertype of all entity types, to all the top-level types, effectively transforming a type forest to a type tree. In Figure 2, we show 3 type ontologies associated with 3 different datasets (see subsection 5.1), with the dummy ENTITY node augmented. We now introduce some notation for referring to aspects of a type tree. The binary relation type z is a subtype of y  is denoted as z < : y . 2 The unique parent of a type y in the type tree is denoted  y  Y , where  y is undefined for y = ENTITY . The immediate subtypes of y (children nodes) are denoted Ch ( y )  Y . Siblings of y , those sharing the same immediate parent, are denoted Sb ( y )  Y , where y (cid:60) Sb ( y ) . In the AIDA FET ontology (see Figure 2), the maximum depth of the tree is L = 3 , and each mention can only be typed with at most 1 type from each level. We term this scenario single-path typing, since there can be only 1 path starting from the root ( ENTITY ) of the type tree. This is in contrast multi-path typing, such as in the BBN dataset, where mentions may be labeled with multiple types on the same level of the tree. Additionally, in AIDA, there are mentions labeled such as as /per/police/<unspecified> . In FIGER, we find instances with labeled type /person but not any further subtype. What does it mean when a mention x is labeled with a partial type path , i.e., a type y but none of the subtypes z < : y ? We consider two interpretations:  Exclusive: x is of type y , but x is not of any type z < : y . We devise different strategies to deal with these two conditions. Under the exclusive case, we add a dummy OTHER node to every intermediate branch node in the type tree. For any mention x labeled with type y but none of the subtypes z < : y , we add this additional label  y /OTHER  to the labels of x (see Figure 2: AIDA). For example, if we interpret a partial type path /person 2 Per programming language literature, e.g. the type system F < : that supports subtyping. in FIGER as exclusive , we add another type /person/OTHER to that instance. Under the undefined case, we do not modify the labels in the dataset. We will see this can make a significant difference depending on the way a specific dataset is annotated. Hidden representations for entity mentions in sentence w are generated by leveraging recent advances in language model pre-training, e.g. ELMo (Peters et al., 2018). 3 The ELMo representation for each token w i is denoted as w i  R d w . Dropout is applied with probability p D to the ELMo vectors. Our mention encoder largely follows Lin and Ji (2019). First a mention representation is derived using the representations of the words in the mention. We apply a max pooling layer atop the mention after a linear transformation: 4 m = MaxPool ( Tw l ,    , Tw r )  R d w . Then we employ mention-to-context attention first described in Zhang et al. (2018) and later employed by Lin and Ji (2019): a context vector c is generated by attending the sentence with a query vector derived from the mention vector m . We use the multiplicative attention of Luong et al. (2015): a i  exp ( m T Qw i ) (2) c = N (cid:88) i = 1 a i w i  R d w (3) The final representation for an entity mention is generated via concatenation of the mention and context vector: [ m ; c ]  R 2 d w . We learn a type embedding y  R d t for each type y  Y . To score an instance with representation [ m ; c ] , we pass it through a 2-layer feed-forward network that maps into the same space as the type space R d t , with tanh as the nonlinearity. The final 3 Lin and Ji (2019) found that ELMo performs better than BERT (Devlin et al., 2019) for FET. Our internal experiments also confirm this finding. We hypothesize that this is due to the richer character-level information contained in lowerlevel ELMo representations that are useful for FET. 4 Lin and Ji (2019) proposed an attentive pooler with a learned global query vector. We found out that a simple max pooling layer achieves similar performance. score is an inner product between the transformed feature vector and the type embedding: F ( x , y ) = FFNN ([ m ; c ])  y . We introduce our novel hierarchical learning-to-rank loss that (1) allows for natural multi-label classification and (2) takes the hierarchical ontology into account. We start with a multi-class hinge loss that ranks positive types above negative types (Weston and Watkins, 1999): J flat ( x , Y ) = (cid:88) y  Y (cid:88) y (cid:48) (cid:60) Y [   F ( x , y ) + F ( x , y (cid:48) )] + (5) where [ x ] + = max { 0 , x } . This is actually learning-to-rank with a ranking SVM (Joachims, 2002): the model learns to rank the positive types y  Y higher than those negative types y (cid:48) (cid:60) Y , by imposing a margin  between y and y (cid:48) : type y should rank higher than y (cid:48) by  . Note that in Equation 5, since it is a linear SVM, the margin hyperparameter  could be just set as 1 (the type embeddings are linearly scalable), and we rely on L 2 regularization to constrain the type embeddings. Multi-level Margins However, this method considers all candidate types to be flat instead of hierarchical  all types are given the same treatment without any prior on their relative position in the type hierarchy. Intuitively, coarser types (higher in the hierarchy) should be easier to determine (e.g. /person vs /location should be fairly easy for the model), but fine-grained types (e.g. /person/artist/singer ) are harder. We encode this intuition by (i) learning to rank types only on the same level in the type tree; (ii) setting different margin parameters for the ranking model with respect to different levels: (cid:88) y  Y (cid:88) y (cid:48)  Sb ( y )\ Y [  lev ( y )  F ( x , y ) + F ( x , y (cid:48) )] + (6) Here lev ( y ) is the level of the type y : for example, lev ( /location ) = 1 , and lev ( /person/artist/singer ) = 3 . In Equation 6, each positive type y is only compared against its negative siblings Sb ( y )\ Y , and the margin hyperparameter is set to be  lev ( y ) , i.e., a margin dependent on which level y is in the tree. Intuitively, we should set  1 >  2 >  3 since our ( 1   )  2 <latexit sha1_base64="Yckm0lgEuwguyWYDTSkkfyWZpZs=">AAACw3icdVFNa9tAEF2rTZq6Sey0x15ETcAtjZFCIDkGSqHHFOokYBkzWo3ixfshdkeNjdAvSY/Jj+q/6doxxUragYXHm3k7b2bSQgpHUfS7Fbx4ubX9aud1+83u3n6ne/D20pnSchxyI429TsGhFBqHJEjidWERVCrxKp19WeavfqJ1wugftChwrOBGi1xwIE9Nup1+fJSALKbwMZmLyfGk24sG0SrC5yBegx5bx8XkoPUryQwvFWriEpwbxVFB4wosCS6xbielwwL4DG5w5KEGhW5crZzX4aFnsjA31j9N4YrdVFSgnFuo1FcqoKl7mluS/8qNSsrPxpXQRUmo+WOjvJQhmXC5hjATFjnJhQfArfBeQz4FC5z8stqHm20cB4kWZd2kpUjRz6ixyY/+8p+9O+mVpvDzaryl+cptuzEDCb+O5pL+I9tUpcbMCFIv9L9Z9FXcKAU6+5QQZZhDKamiOVHt7xk/vd5zcHk8iKNB/P2kd95fX3aHvWcfWJ/F7JSds2/sgg0ZZyW7Y/fsIfgazAIb0GNp0Fpr3rFGBPUfKwDgvg==</latexit> <latexit sha1_base64="Yckm0lgEuwguyWYDTSkkfyWZpZs=">AAACw3icdVFNa9tAEF2rTZq6Sey0x15ETcAtjZFCIDkGSqHHFOokYBkzWo3ixfshdkeNjdAvSY/Jj+q/6doxxUragYXHm3k7b2bSQgpHUfS7Fbx4ubX9aud1+83u3n6ne/D20pnSchxyI429TsGhFBqHJEjidWERVCrxKp19WeavfqJ1wugftChwrOBGi1xwIE9Nup1+fJSALKbwMZmLyfGk24sG0SrC5yBegx5bx8XkoPUryQwvFWriEpwbxVFB4wosCS6xbielwwL4DG5w5KEGhW5crZzX4aFnsjA31j9N4YrdVFSgnFuo1FcqoKl7mluS/8qNSsrPxpXQRUmo+WOjvJQhmXC5hjATFjnJhQfArfBeQz4FC5z8stqHm20cB4kWZd2kpUjRz6ixyY/+8p+9O+mVpvDzaryl+cptuzEDCb+O5pL+I9tUpcbMCFIv9L9Z9FXcKAU6+5QQZZhDKamiOVHt7xk/vd5zcHk8iKNB/P2kd95fX3aHvWcfWJ/F7JSds2/sgg0ZZyW7Y/fsIfgazAIb0GNp0Fpr3rFGBPUfKwDgvg==</latexit> <latexit sha1_base64="Yckm0lgEuwguyWYDTSkkfyWZpZs=">AAACw3icdVFNa9tAEF2rTZq6Sey0x15ETcAtjZFCIDkGSqHHFOokYBkzWo3ixfshdkeNjdAvSY/Jj+q/6doxxUragYXHm3k7b2bSQgpHUfS7Fbx4ubX9aud1+83u3n6ne/D20pnSchxyI429TsGhFBqHJEjidWERVCrxKp19WeavfqJ1wugftChwrOBGi1xwIE9Nup1+fJSALKbwMZmLyfGk24sG0SrC5yBegx5bx8XkoPUryQwvFWriEpwbxVFB4wosCS6xbielwwL4DG5w5KEGhW5crZzX4aFnsjA31j9N4YrdVFSgnFuo1FcqoKl7mluS/8qNSsrPxpXQRUmo+WOjvJQhmXC5hjATFjnJhQfArfBeQz4FC5z8stqHm20cB4kWZd2kpUjRz6ixyY/+8p+9O+mVpvDzaryl+cptuzEDCb+O5pL+I9tUpcbMCFIv9L9Z9FXcKAU6+5QQZZhDKamiOVHt7xk/vd5zcHk8iKNB/P2kd95fX3aHvWcfWJ/F7JSds2/sgg0ZZyW7Y/fsIfgazAIb0GNp0Fpr3rFGBPUfKwDgvg==</latexit> <latexit sha1_base64="Yckm0lgEuwguyWYDTSkkfyWZpZs=">AAACw3icdVFNa9tAEF2rTZq6Sey0x15ETcAtjZFCIDkGSqHHFOokYBkzWo3ixfshdkeNjdAvSY/Jj+q/6doxxUragYXHm3k7b2bSQgpHUfS7Fbx4ubX9aud1+83u3n6ne/D20pnSchxyI429TsGhFBqHJEjidWERVCrxKp19WeavfqJ1wugftChwrOBGi1xwIE9Nup1+fJSALKbwMZmLyfGk24sG0SrC5yBegx5bx8XkoPUryQwvFWriEpwbxVFB4wosCS6xbielwwL4DG5w5KEGhW5crZzX4aFnsjA31j9N4YrdVFSgnFuo1FcqoKl7mluS/8qNSsrPxpXQRUmo+WOjvJQhmXC5hjATFjnJhQfArfBeQz4FC5z8stqHm20cB4kWZd2kpUjRz6ixyY/+8p+9O+mVpvDzaryl+cptuzEDCb+O5pL+I9tUpcbMCFIv9L9Z9FXcKAU6+5QQZZhDKamiOVHt7xk/vd5zcHk8iKNB/P2kd95fX3aHvWcfWJ/F7JSds2/sgg0ZZyW7Y/fsIfgazAIb0GNp0Fpr3rFGBPUfKwDgvg==</latexit>  2 <latexit sha1_base64="e6geauY33jiuodw/0nXuPUGwUKo=">AAACvXicdVHbattAEF0rvaTuLUkf+yJqAqEUI4VA+9ZAX/KYQp0EJGNGq1G8eC9id9TYCH9G30rzXf2bjh1TrKQdWDicmbNzZqaotQqUJL970c6jx0+e7j7rP3/x8tXrvf2Di+AaL3EknXb+qoCAWlkckSKNV7VHMIXGy2L2ZZW//I4+KGe/0aLGsYFrqyolgZjKctD1FPK5mhxP9gbJMFlH/BCkGzAQmzif7Pd+5qWTjUFLUkMIWZrUNG7Bk5Ial/28CViDnME1ZgwtGAzjdu15GR8yU8aV8/wsxWt2W9GCCWFhCq40QNNwP7ci/5XLGqo+jVtl64bQyrtGVaNjcvFqAXGpPErSCwYgvWKvsZyCB0m8pv7hdpsgQaNHvezSWhXIM1rs8tlf/gO706x0Nc9r8Ybma7f9zgykeB3dJf1Htq0qnJsRFCzk3zxylXTGgC3f50QlVtBoamlOtOR7pvev9xBcHA/TZJh+PRmcHm0uuyveinfiSKTiozgVZ+JcjIQUTvwQv8Rt9DnCSEf2rjTqbTRvRCeimz9leN+2</latexit> <latexit sha1_base64="e6geauY33jiuodw/0nXuPUGwUKo=">AAACvXicdVHbattAEF0rvaTuLUkf+yJqAqEUI4VA+9ZAX/KYQp0EJGNGq1G8eC9id9TYCH9G30rzXf2bjh1TrKQdWDicmbNzZqaotQqUJL970c6jx0+e7j7rP3/x8tXrvf2Di+AaL3EknXb+qoCAWlkckSKNV7VHMIXGy2L2ZZW//I4+KGe/0aLGsYFrqyolgZjKctD1FPK5mhxP9gbJMFlH/BCkGzAQmzif7Pd+5qWTjUFLUkMIWZrUNG7Bk5Ial/28CViDnME1ZgwtGAzjdu15GR8yU8aV8/wsxWt2W9GCCWFhCq40QNNwP7ci/5XLGqo+jVtl64bQyrtGVaNjcvFqAXGpPErSCwYgvWKvsZyCB0m8pv7hdpsgQaNHvezSWhXIM1rs8tlf/gO706x0Nc9r8Ybma7f9zgykeB3dJf1Htq0qnJsRFCzk3zxylXTGgC3f50QlVtBoamlOtOR7pvev9xBcHA/TZJh+PRmcHm0uuyveinfiSKTiozgVZ+JcjIQUTvwQv8Rt9DnCSEf2rjTqbTRvRCeimz9leN+2</latexit> <latexit sha1_base64="e6geauY33jiuodw/0nXuPUGwUKo=">AAACvXicdVHbattAEF0rvaTuLUkf+yJqAqEUI4VA+9ZAX/KYQp0EJGNGq1G8eC9id9TYCH9G30rzXf2bjh1TrKQdWDicmbNzZqaotQqUJL970c6jx0+e7j7rP3/x8tXrvf2Di+AaL3EknXb+qoCAWlkckSKNV7VHMIXGy2L2ZZW//I4+KGe/0aLGsYFrqyolgZjKctD1FPK5mhxP9gbJMFlH/BCkGzAQmzif7Pd+5qWTjUFLUkMIWZrUNG7Bk5Ial/28CViDnME1ZgwtGAzjdu15GR8yU8aV8/wsxWt2W9GCCWFhCq40QNNwP7ci/5XLGqo+jVtl64bQyrtGVaNjcvFqAXGpPErSCwYgvWKvsZyCB0m8pv7hdpsgQaNHvezSWhXIM1rs8tlf/gO706x0Nc9r8Ybma7f9zgykeB3dJf1Htq0qnJsRFCzk3zxylXTGgC3f50QlVtBoamlOtOR7pvev9xBcHA/TZJh+PRmcHm0uuyveinfiSKTiozgVZ+JcjIQUTvwQv8Rt9DnCSEf2rjTqbTRvRCeimz9leN+2</latexit> <latexit sha1_base64="e6geauY33jiuodw/0nXuPUGwUKo=">AAACvXicdVHbattAEF0rvaTuLUkf+yJqAqEUI4VA+9ZAX/KYQp0EJGNGq1G8eC9id9TYCH9G30rzXf2bjh1TrKQdWDicmbNzZqaotQqUJL970c6jx0+e7j7rP3/x8tXrvf2Di+AaL3EknXb+qoCAWlkckSKNV7VHMIXGy2L2ZZW//I4+KGe/0aLGsYFrqyolgZjKctD1FPK5mhxP9gbJMFlH/BCkGzAQmzif7Pd+5qWTjUFLUkMIWZrUNG7Bk5Ial/28CViDnME1ZgwtGAzjdu15GR8yU8aV8/wsxWt2W9GCCWFhCq40QNNwP7ci/5XLGqo+jVtl64bQyrtGVaNjcvFqAXGpPErSCwYgvWKvsZyCB0m8pv7hdpsgQaNHvezSWhXIM1rs8tlf/gO706x0Nc9r8Ybma7f9zgykeB3dJf1Htq0qnJsRFCzk3zxylXTGgC3f50QlVtBoamlOtOR7pvev9xBcHA/TZJh+PRmcHm0uuyveinfiSKTiozgVZ+JcjIQUTvwQv8Rt9DnCSEf2rjTqbTRvRCeimz9leN+2</latexit>  1 <latexit sha1_base64="mg5btP8pvA4PX4Lgss3gJW+5gB0=">AAACvXicdVHbattAEF0rbZO6l1z62BdREwilGKkU0rcE+tLHFOokIBkzWo3ixXsRu6PGRvgz+lba7+rfdOyYYiXtwMLhzJydMzNFrVWgJPndi3YePX6yu/e0/+z5i5f7B4dHl8E1XuJIOu38dQEBtbI4IkUar2uPYAqNV8Xs0yp/9Q19UM5+pUWNYwM3VlVKAjGV5aDrKeRzNUknB4NkmKwjfgjSDRiITVxMDns/8tLJxqAlqSGELE1qGrfgSUmNy37eBKxBzuAGM4YWDIZxu/a8jI+ZKePKeX6W4jW7rWjBhLAwBVcaoGm4n1uR/8plDVUfx62ydUNo5V2jqtExuXi1gLhUHiXpBQOQXrHXWE7BgyReU/94u02QoNGjXnZprQrkGS12+ewv/47daVa6mue1eEvztdt+ZwZSvI7ukv4j21YVzs0IChbybx65SjpjwJZvc6ISK2g0tTQnWvI90/vXewgu3w/TZJh++TA4P9lcdk+8Fm/EiUjFqTgXn8WFGAkpnPgufopf0VmEkY7sXWnU22heiU5Et38AYynftQ==</latexit> <latexit sha1_base64="mg5btP8pvA4PX4Lgss3gJW+5gB0=">AAACvXicdVHbattAEF0rbZO6l1z62BdREwilGKkU0rcE+tLHFOokIBkzWo3ixXsRu6PGRvgz+lba7+rfdOyYYiXtwMLhzJydMzNFrVWgJPndi3YePX6yu/e0/+z5i5f7B4dHl8E1XuJIOu38dQEBtbI4IkUar2uPYAqNV8Xs0yp/9Q19UM5+pUWNYwM3VlVKAjGV5aDrKeRzNUknB4NkmKwjfgjSDRiITVxMDns/8tLJxqAlqSGELE1qGrfgSUmNy37eBKxBzuAGM4YWDIZxu/a8jI+ZKePKeX6W4jW7rWjBhLAwBVcaoGm4n1uR/8plDVUfx62ydUNo5V2jqtExuXi1gLhUHiXpBQOQXrHXWE7BgyReU/94u02QoNGjXnZprQrkGS12+ewv/47daVa6mue1eEvztdt+ZwZSvI7ukv4j21YVzs0IChbybx65SjpjwJZvc6ISK2g0tTQnWvI90/vXewgu3w/TZJh++TA4P9lcdk+8Fm/EiUjFqTgXn8WFGAkpnPgufopf0VmEkY7sXWnU22heiU5Et38AYynftQ==</latexit> <latexit sha1_base64="mg5btP8pvA4PX4Lgss3gJW+5gB0=">AAACvXicdVHbattAEF0rbZO6l1z62BdREwilGKkU0rcE+tLHFOokIBkzWo3ixXsRu6PGRvgz+lba7+rfdOyYYiXtwMLhzJydMzNFrVWgJPndi3YePX6yu/e0/+z5i5f7B4dHl8E1XuJIOu38dQEBtbI4IkUar2uPYAqNV8Xs0yp/9Q19UM5+pUWNYwM3VlVKAjGV5aDrKeRzNUknB4NkmKwjfgjSDRiITVxMDns/8tLJxqAlqSGELE1qGrfgSUmNy37eBKxBzuAGM4YWDIZxu/a8jI+ZKePKeX6W4jW7rWjBhLAwBVcaoGm4n1uR/8plDVUfx62ydUNo5V2jqtExuXi1gLhUHiXpBQOQXrHXWE7BgyReU/94u02QoNGjXnZprQrkGS12+ewv/47daVa6mue1eEvztdt+ZwZSvI7ukv4j21YVzs0IChbybx65SjpjwJZvc6ISK2g0tTQnWvI90/vXewgu3w/TZJh++TA4P9lcdk+8Fm/EiUjFqTgXn8WFGAkpnPgufopf0VmEkY7sXWnU22heiU5Et38AYynftQ==</latexit> <latexit sha1_base64="mg5btP8pvA4PX4Lgss3gJW+5gB0=">AAACvXicdVHbattAEF0rbZO6l1z62BdREwilGKkU0rcE+tLHFOokIBkzWo3ixXsRu6PGRvgz+lba7+rfdOyYYiXtwMLhzJydMzNFrVWgJPndi3YePX6yu/e0/+z5i5f7B4dHl8E1XuJIOu38dQEBtbI4IkUar2uPYAqNV8Xs0yp/9Q19UM5+pUWNYwM3VlVKAjGV5aDrKeRzNUknB4NkmKwjfgjSDRiITVxMDns/8tLJxqAlqSGELE1qGrfgSUmNy37eBKxBzuAGM4YWDIZxu/a8jI+ZKePKeX6W4jW7rWjBhLAwBVcaoGm4n1uR/8plDVUfx62ydUNo5V2jqtExuXi1gLhUHiXpBQOQXrHXWE7BgyReU/94u02QoNGjXnZprQrkGS12+ewv/47daVa6mue1eEvztdt+ZwZSvI7ukv4j21YVzs0IChbybx65SjpjwJZvc6ISK2g0tTQnWvI90/vXewgu3w/TZJh++TA4P9lcdk+8Fm/EiUjFqTgXn8WFGAkpnPgufopf0VmEkY7sXWnU22heiU5Et38AYynftQ==</latexit> ( 1   )  1 <latexit sha1_base64="NQaqpUnq6dCEYoNBLkI0fjNzoB0=">AAACw3icdVHbihNBEO2MtzVeNquPvgyGhSgaZkTQxwURfFzB7C5kQqjpqdk06cvQXaMJw3yJPupH+TdWskEyu1rQcDhVp+tUVV5pFShJfveiW7fv3L13cL//4OGjx4eDoydnwdVe4kQ67fxFDgG1sjghRRovKo9gco3n+fLDJn/+FX1Qzn6hdYUzA5dWlUoCMTUfHI7S1xnoagEvspWap/PBMBkn24hvgnQHhmIXp/Oj3o+scLI2aElqCGGaJhXNGvCkpMa2n9UBK5BLuMQpQwsGw6zZOm/jY2aKuHSen6V4y+4rGjAhrE3OlQZoEa7nNuS/ctOayvezRtmqJrTyqlFZ65hcvFlDXCiPkvSaAUiv2GssF+BBEi+rf7zfJkjQ6FG3XVqrHHlGi11++pd/xe40K13F81r8Rqut235nBlK8ju6S/iPbV+XOLQlyFvJvHrlKOmPAFi8zogJLqDU1tCJq+Z7p9evdBGdvxmkyTj+/HZ6Mdpc9EM/EczESqXgnTsQncSomQopafBc/xa/oY7SMfERXpVFvp3kqOhG1fwAoseC9</latexit> <latexit sha1_base64="NQaqpUnq6dCEYoNBLkI0fjNzoB0=">AAACw3icdVHbihNBEO2MtzVeNquPvgyGhSgaZkTQxwURfFzB7C5kQqjpqdk06cvQXaMJw3yJPupH+TdWskEyu1rQcDhVp+tUVV5pFShJfveiW7fv3L13cL//4OGjx4eDoydnwdVe4kQ67fxFDgG1sjghRRovKo9gco3n+fLDJn/+FX1Qzn6hdYUzA5dWlUoCMTUfHI7S1xnoagEvspWap/PBMBkn24hvgnQHhmIXp/Oj3o+scLI2aElqCGGaJhXNGvCkpMa2n9UBK5BLuMQpQwsGw6zZOm/jY2aKuHSen6V4y+4rGjAhrE3OlQZoEa7nNuS/ctOayvezRtmqJrTyqlFZ65hcvFlDXCiPkvSaAUiv2GssF+BBEi+rf7zfJkjQ6FG3XVqrHHlGi11++pd/xe40K13F81r8Rqut235nBlK8ju6S/iPbV+XOLQlyFvJvHrlKOmPAFi8zogJLqDU1tCJq+Z7p9evdBGdvxmkyTj+/HZ6Mdpc9EM/EczESqXgnTsQncSomQopafBc/xa/oY7SMfERXpVFvp3kqOhG1fwAoseC9</latexit> <latexit sha1_base64="NQaqpUnq6dCEYoNBLkI0fjNzoB0=">AAACw3icdVHbihNBEO2MtzVeNquPvgyGhSgaZkTQxwURfFzB7C5kQqjpqdk06cvQXaMJw3yJPupH+TdWskEyu1rQcDhVp+tUVV5pFShJfveiW7fv3L13cL//4OGjx4eDoydnwdVe4kQ67fxFDgG1sjghRRovKo9gco3n+fLDJn/+FX1Qzn6hdYUzA5dWlUoCMTUfHI7S1xnoagEvspWap/PBMBkn24hvgnQHhmIXp/Oj3o+scLI2aElqCGGaJhXNGvCkpMa2n9UBK5BLuMQpQwsGw6zZOm/jY2aKuHSen6V4y+4rGjAhrE3OlQZoEa7nNuS/ctOayvezRtmqJrTyqlFZ65hcvFlDXCiPkvSaAUiv2GssF+BBEi+rf7zfJkjQ6FG3XVqrHHlGi11++pd/xe40K13F81r8Rqut235nBlK8ju6S/iPbV+XOLQlyFvJvHrlKOmPAFi8zogJLqDU1tCJq+Z7p9evdBGdvxmkyTj+/HZ6Mdpc9EM/EczESqXgnTsQncSomQopafBc/xa/oY7SMfERXpVFvp3kqOhG1fwAoseC9</latexit> <latexit sha1_base64="NQaqpUnq6dCEYoNBLkI0fjNzoB0=">AAACw3icdVHbihNBEO2MtzVeNquPvgyGhSgaZkTQxwURfFzB7C5kQqjpqdk06cvQXaMJw3yJPupH+TdWskEyu1rQcDhVp+tUVV5pFShJfveiW7fv3L13cL//4OGjx4eDoydnwdVe4kQ67fxFDgG1sjghRRovKo9gco3n+fLDJn/+FX1Qzn6hdYUzA5dWlUoCMTUfHI7S1xnoagEvspWap/PBMBkn24hvgnQHhmIXp/Oj3o+scLI2aElqCGGaJhXNGvCkpMa2n9UBK5BLuMQpQwsGw6zZOm/jY2aKuHSen6V4y+4rGjAhrE3OlQZoEa7nNuS/ctOayvezRtmqJrTyqlFZ65hcvFlDXCiPkvSaAUiv2GssF+BBEi+rf7zfJkjQ6FG3XVqrHHlGi11++pd/xe40K13F81r8Rqut235nBlK8ju6S/iPbV+XOLQlyFvJvHrlKOmPAFi8zogJLqDU1tCJq+Z7p9evdBGdvxmkyTj+/HZ6Mdpc9EM/EczESqXgnTsQncSomQopafBc/xa/oY7SMfERXpVFvp3kqOhG1fwAoseC9</latexit> Figure 3: Hierarchical learning-to-rank. model should be able to learn a larger margin between easier pairs: we show that this is superior than using a single margin in our experiments. Analogous to the reasoning that in Equation 5 the margin  can just be 1, only the relative ratios between  's are important. For simplicity, 5 if the ontology has L levels, we assign  l = L  l + 1 . Flexible Threshold Equation 6 only ranks positive types higher than negative types so that all children types given a parent type are ranked based on their relevance to the entity mention. What should be the threshold between positive and negative types? We could set the threshold to be 0 (ap-proaching the multi-label classification problem as a set of binary classification problem, see Lin and Ji (2019)), or tune an adaptive, type-specific threshold for each parent type (Zhang et al., 2018). Here, we propose a simpler method. 5 We did hyperparameter search on these margin hyperparameters and found that Equation 7 generalized well. parent type /person/artist can be considered as a kind of prior for all types of artists, the model should learn that the positive type singer should have a higher confidence than artist, and in turn, higher than other types of artists like author or actor. Hence the ranker should learn that a positive subtype should rank higher than its parent, and its parent should rank higher than its negative children. Under this formulation, at decoding time, given parent type y , a child subtype z < : y that scores higher than y should be output as a positive label. We translate the ranking relation in Equation 8 into a ranking loss that extends Equation 6. In Equation 6, there is an expected margin  between positive types and negative types. Since we inserted the parent in the middle, we divide the margin  into  and ( 1   )  :  being the margin between positive types and the parent; and ( 1   )  is the margin between the parent and the negative types. For a visualization see Figure 3. The hyperparameter   [ 0 , 1 ] can be used to tune the precision-recall tradeoff when outputting types: the smaller  , the smaller the expected margin there is between positive types and the parent. This intuitively increases precision but decreases recall (only very confident types can be output). Vice versa, increasing  decreases precision but increase recall. Therefore we learn 3 sets of ranking relations from Equation 8: (i) positive types should be scored above parent by  ; (ii) parent should be scored above any negative sibling types by ( 1   )  ; (iii) positive types should be scored above negative sibling types by  . Our final hierarchical ranking loss is formulated as follows. J y (cid:31)  y = [  lev ( y )  F ( x , y ) + F ( x ,  y )] + J  y (cid:31) y (cid:48) = (cid:88) y (cid:48)  Sb ( y )\ Y [( 1   )  lev ( y )  F ( x ,  y ) + F ( x , y (cid:48) )] + J y (cid:31) y (cid:48) = (cid:88) y (cid:48)  Sb ( y )\ Y [  lev ( y )  F ( x , y ) + F ( x , y (cid:48) )] + J hier ( x , Y ) = (cid:88) y  Y (cid:0) J y (cid:31)  y + J  y (cid:31) y (cid:48) + J y (cid:31) y (cid:48) (cid:1) (9) 4.4 Decoding Predicting the types for each entity mention can be performed via iterative searching on the type tree, from the root ENTITY node to coarser types, then to finer-grained types. This ensures that our output does not violate the hierarchical property, i.e., if a subtype is output, its parent must be output. Given instance x we compute the score F ( x , y ) for each type y  Y , the searching process starts with the root node ENTITY of the type tree in the queue. For each type y in the node, a child node z < : y (subtypes) is added to the predicted type set if F ( x , z ) > F ( x , y ) , corresponding to the ranking relation in Equation 8 that the model has learned. 6 Here we only take the topk element to add to the queue to prevent from over-generating types. This can also be used to enforce the single-path property (setting k = 1) if the dataset is single-path. For each level i in the type hierarchy, we limit the branching factor (allowed children) to be k i . The algorithm is listed in Algorithm 1, where the function TOPK ( S , k , f ) selects the topk elements from S with respect to the function f . Each type y  Y in the ontology is assigned a type embedding y  R d t . We notice the binary subtyping relation  < :   Y  Y on the types. Trouillon et al. (2016) proposed the relation embedding method ComplEx that works well with anti-symmetric and transitive relations such as subtyping. It has been employed in FET before 6 For the OntoNotes dataset, we introduce another set of per-level hyperparameters  lev ( y ) , and the threshold value F ( x , y ) is modified to F ( x , y ) +  lev ( y ) , akin to the adaptive threshold in Zhang et al. (2018). This is due to a large type distribution mismatch between the training and dev/test sets in OntoNotes (in dev/test there are a lot of instances with the single type /other but not in the training set). For other datasets they are unused, i.e. just 0.  in Murty et al. (2018), ComplEx is added to the loss to regulate the type embeddings. ComplEx operates in the complex space  we use the natural isomorphism between real and complex spaces to map the type embedding into complex space (first half of the embedding vector as the real part, and the second half as the imaginary part):  : R d t  C d t / 2 (10) t = [ Re  ( t ) ; Im  ( t ) ] (11) We learn a single relation embedding r  C d t / 2 for the subtyping relation. Given type y and z , the subtyping statement y < : z is modeled using the following scoring function: r ( y , z ) = Re (cid:16) r  (cid:16)  ( y ) (cid:12)  ( z ) (cid:17)(cid:17) (12) where (cid:12) is element-wise product and x is the complex conjugate of x . If y < : z then r ( y , z ) > 0 ; and vice versa, r ( y , z ) < 0 if y  : z . Loss Given instance ( x , Y ) , for each positive type y  Y , we learn the following relations: y < :  y y  : y (cid:48) ,  y (cid:48)  Sb ( y ) y  : y (cid:48) ,  y (cid:48)  Sb (  y ) (13) Translating these relation constraints as a binary classification problem (is or is not a subtype) under a primal SVM, we get a hinge loss: J rel ( x , Y ) = (cid:88) y  Y (cid:32) [ 1  r ( y ,  y )] + + (cid:88) y (cid:48)  Sb ( y ) Sb (  y ) [ 1 + r ( y , y (cid:48) )] + (cid:33) . (14) This is different from Murty et al. (2018), where a binary cross-entropy loss on randomly sampled ( y , y (cid:48) ) pairs is used. Our experiments showed that the loss in Equation 14 performs better than the cross-entropy version, due to the structure of the training pairs: we use siblings and siblings of parents as negative samples (these are types closer to the positive parent type), hence are training with more competitive negative samples. The AdamW optimizer (Loshchilov and Hutter, 2019) is used to train the model, as it is shown to be superior than the original Adam under L 2 regularization. Hyperparameters  (ratio of margin above/below threshold),  (weight of subtyping relation constraint), and  ( L 2 regularization coefficient) are tuned. At validation time, we tune the maximum branching factors for each level k 1 ,    , k L . 7 These parameters tune the trade-off between the precision and recall for each layer and prevents over-generation (as we observed in some cases). All hyperparameters are tuned so that models achieve maximum micro F 1 scores (see subsection 5.4). AIDA The AIDA Phase 1 practice dataset for hierarchical entity typing comprises of 297 documents from LDC2019E04 / LDC2019E07 , and the evaluation dataset is from LDC2019E42 / LDC2019E77 . We take only the English part of the data, and use the practice dataset as train/dev, and the evaluation dataset as test. The practice dataset comprises of 3 domains, labeled as R103 , R105 , and R107 . Since the evaluation dataset is out-of-domain, we use the smallest domain R105 as dev, and the remaining R103 and R107 as train. The AIDA entity dataset has a 3-level ontology, termed type , subtype , and subsubtype . A mention can only have one label for each level, hence the dataset is single-path , thus the branching factors ( k 1 , k 2 , k 3 ) for the three layers are set to ( 1 , 1 , 1 ) . BBN Weischedel and Brunstein (2005) labeled a portion of the one million word Penn Treebank corpus of Wall Street Journal texts ( LDC95T7 ) using a two-level hierarchy, resulting in the BBN Pronoun Coreference and Entity Type Corpus. We follow the train/test split by Ren et al. (2016b), and follow the train/dev split by Zhang et al. (2018). OntoNotes Gillick et al. (2014) sampled sentences from the OntoNotes corpus and annotated the entities using 89 types. We follow the train/dev/test data split by Shimaoka et al. (2017). FIGER Ling and Weld (2012) sampled a dataset from Wikipdia articles and news reports. Entity mentions in these texts are mapped to a 113-type ontology derived from Freebase (Bollacker et al., 2008). Again, we follow the data split by Shimaoka et al. (2017). To best compare to recent prior work, we follow Lin and Ji (2019) where the ELMo encodings of words are fixed and not updated. We use all 3 layers of ELMo output, so the initial embedding has dimension d w = 3072 . We set the type embedding dimensionality to be d t = 1024 . The initial learning rate is 10  5 and the batch size is 256. Hyperparameter choices are tuned on dev sets, and are listed in Table 1. We employ early stopping: choosing the model that yields the best micro F 1 score on dev sets. Our models are implemented using AllenNLP (Gardner et al., 2018), with implementation for subtyping relation constraints from OpenKE (Han et al., 2018). We compare our approach to major prior work in FET that are capable of multi-path entity typing. 9 For AIDA, since there are no prior work on this dataset to our knowledge, we also implemented multi-label classification as set of binary classifier models (similar to Lin and Ji (2019)) as a baseline, with our mention feature extractor. The results are shown in Table 2 as Multi-label. 8 The OntoNotes dataset has an additional set of hyperparameters, i.e. the per-level threshold  1 , 2 , 3 = ( 2 . 5 , 3 . 0 , 0 . 0 ) . 9 Zhang et al. (2018) included document-level information in their best resultsfor fair comparison, we used their results without document context, as are reported in their ablation tests. We follow prior work and use strict accuracy (Acc), macro F 1 (MaF), and micro F 1 (MiF) scores. Given instance x i , we denote the gold type set as Y i and the predicted type set  Y i . The strict accuracy is the ratio of instances where Y i =  Y i . Macro F 1 is the average of all F 1 scores between Y i and  Y i for all instances, whereas micro F 1 counts total true positives, false negatives and false positives globally. We also investigate per-level accuracies on AIDA. The accuracy on level l is the ratio of instances whose predicted type set and gold type set are identical at level l . If there is no type output at level l , we append with OTHER to create a dummy type at level l : e.g. /person/OTHER/OTHER . Hence accuracy of the last level (in AIDA, level 3) is equal to the strict accuracy. All our results are run under the two conditions regarding partial type paths: exclusive or undefined. The result of the AIDA dataset is shown in Table 2. Our model under the exclusive case outperforms a multi-label classification baseline over all metrics. Of the 187 types specified in the AIDA ontology, the train/dev set only covers 93 types. The test set covers 85 types, of which 63 are seen types. We could perform zero-shot entity typing by initializing a type's embedding using the type name (e.g. /fac/structure/plaza ) together with its description (e.g.  An open urban public space, such as a city square ) as is designated in the data annotation manual. We leave this as future work. : Result has document-level context information, hence not comparable. Results for the BBN, OntoNotes, and FIGER can be found in Table 3. Across 3 datasets, our method produces the state-of-the-art performance on strict accuracy and micro F 1 scores, and state-of-the-art or comparable (  0 . 5% ) performance on macro F 1 score, as compared to prior models, e.g. (Lin and Ji, 2019). Especially, our method improves upon the strict accuracy substantially (4% 8%) across these datasets, showing our decoder are better at outputting exact correct type sets. Partial type paths: exclusive or undefined? Interestingly, we found that for AIDA and FIGER, partial type paths should be better considered as exclusive , whereas for BBN and OntoNotes, considering them as undefined leads to better performance. We hypothesize that this comes from how the data is annotatatedthe annotation manual may contain directives as whether to interpret partial type paths as exclusive or undefined, or the data may be non-exhaustively annotated, leading to undefined partial types. We advocate for careful investigation into partial type paths for future experiments and data curation. Ablation Studies We compare our best model with various components of our model removed, to study the gain from each component. From the best of these two settings ( exclusive and undefined ), we report the performance of (i) removing the subtyping constraint as is described in subsection 4.5; (ii) substituting the multi-level margins in Equation 7 with a flat margin, i.e., margins on all levels are set to be 1. These results are shown in Table 2 and Table 3 under our best results, and they show that both multi-level margins and subtyping relation constraints offer orthogonal improvements to our models. Error Analysis We identify common patterns of errors, coupled with typical examples:  Confusing types: In BBN, our model outputs /gpe/city when the gold type is /location/region for  ... in shipments from the Valley of either hardware or software goods.  These types are semantically similar, and our model failed to discriminate between these types. Incomplete types: In FIGER, given instance  ... multi-agency investigation headed by the U.S. Immigration and Customs Enforcement 's homeland security investigations unit , the gold types are /government agency and /organization , but our model failed to output /organization . Focusing on only parts of the mention: In AIDA, given instance  ... suggested they were the work of Russian special forces assassins out to blacken the image of Kievs pro-Western authorities , our model outputs /org/government whereas the gold type is /per/militarypersonnel . Our model focused on the Russian special forces part, but ignored the assassins part. Better mention representation is required to correct this, possibly by introducing type-aware mention representationwe leave this as future work. We proposed (i) a novel multi-level learning to rank loss function that operates on a type tree, and (ii) an accompanying coarse-to-fine decoder to fully embrace the ontological structure of the types for hierarchical entity typing. Our approach achieved state-of-the-art performance across various datasets, and made substantial improvement (48%) upon strict accuracy. Additionally, we advocate for careful investigation into partial type paths : their interpretation relies on how the data is annotated, and in turn, in-fluences typing performance. We thank our colleague Guanghui Qin and the anonymous reviewers for their insightful suggestions and comments. This research benefited from support by the JHU Human Language Technology Center of Excellence (HLTCOE), and DARPA AIDA. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA or the U.S. Government.
objective objective method result abstain abstain other abstain abstain other abstain abstain abstain abstain objective abstain other abstain other method other other objective other other other other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method method abstain abstain abstain abstain method abstain method abstain method method abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method other abstain method abstain method abstain method abstain abstain abstain abstain method abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method method abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain objective abstain abstain result abstain other other other other