Training neural models for named entity recognition (NER) in a new domain often requires additional human annotations that are usually expensive and time-consuming to collect. Thus, a crucial research question is how to obtain supervision in a cost-effective way. In this paper, we introduce entity triggers, an effective proxy of human explanations for facilitating label-efficient learning of NER models. An entity trigger is defined as a group of words in a sentence that helps to explain why humans would recognize an entity in the sentence. We crowd-sourced 14k entity triggers for two well-studied NER datasets 1 . Our proposed model, Trigger Matching Network , jointly learns trigger representations and soft matching module with self-attention such that can generalize to unseen sentences easily for tagging. The framework is significantly more cost-effective than the traditional frameworks. Named entity recognition (NER) is a fundamental information extraction task that focuses on extracting entities from a given text and classifying them using pre-defined categories (e.g., persons, locations, organizations) (Nadeau and Sekine, 2007). Recent advances in NER have primarily focused on training neural network models with an abundance of human annotations, yielding state-of-the-art results (Lample et al., 2016). However, collecting human annotations for NER is expensive and time-consuming, especially in social media messages (Lin et al., 2017a) and technical domains such as biomedical publications, financial documents, legal reports, etc. As we seek to advance NER into more domains with less human effort,  The first two authors contributed equally. how to learn neural models for NER in a cost-effective way becomes a crucial research problem. The standard protocol for obtaining an annotated NER dataset involves an annotator selecting token spans in a sentence as mentions of entities, and labeling them with an entity type. However, such annotation process provides limited supervision per example . Consequently, one would need large amount of annotations in order to train high-performing models for a broad range of entity types, which can clearly be cost-prohibitive. The key question is then how can we learn an effective NER model in presence of limited quantities of labeled data ? We, as humans, recognize an entity within a sentence based on certain words or phrases that act as cues. For instance, we could infer that Kasd-frcxzv' is likely to be a location entity in the sentence  Tom traveled a lot last year in Kasdfrcxzv.  We recognize this entity because of the cue phrase  travel ... in , which suggests there should be a location entity following the word 'in'. We call such phrases entity triggers. Similar to the way these triggers guide our recognition process, we hypothesize that they can also help the model to learn to generalize efficiently. Specifically, we define an entity trigger (or trigger for simplicity) as a group of words that can help explain the recognition process of a particular entity in the same sentence. For example, in Figure 1, had ... lunch at 2 and where the food are two distinct triggers associated with the RESTAURANT entity Rumble Fish. An entity trigger should be a necessary and sufficient cue for humans to recognize its associated entity even if we mask the entity with a random word. Thus, unnecessary words such as  fantastic  should not be considered part of the entity trigger. In this paper, we argue that a combination of entity triggers and standard entity annotations can enhance the generalization power of NER models. This approach is more powerful because unlabeled sentences, such as Bill enjoyed a great dinner with Alice at Zcxlbz. , can be matched with the existing trigger had ... lunch at via their semantic relatedness. This makes it easier for a model to recognize Zcxlbz as a RESTAURANT entity. In contrast, if we only have the entity annotation itself (i.e., Rumble Fish ) as supervision, the model will require many similar examples in order to learn this simple pattern. We hypothesize that using triggers as additional supervision is a more cost-effective way to train models. We crowd-sourced annotations of 14,708 triggers on two well-studied NER datasets to study their usefulness for the NER task. Also, we propose a novel framework named Trigger Matching Network that learns trigger representations indicative of entity types during the training phase, and identifies triggers in an unlabeled sentence at inference time to guide a traditional entity tagger for delivering better overall NER performance. Different from conventional training, our learning process has two stages, where the first stage comprises jointly training a trigger classifier and the semantic trigger matcher, followed by a second stage that leverages the trigger representation and the encoding of the given sentence using an attention mechanism to learn a tagger. Experiments show that the proposed model using only 20% of the trigger-annotated sentences results in a comparable performance as using 70% of conventional annotated sentences. 2 Problem Formulation We consider the problem of how to cost-effectively learn a model for NER using entity triggers . In this 2 Note that a trigger can be a discontinuous phrase. section, we introduce basic concepts and their notations, present the conventional data annotation process for NER, and provide a formal task defini-tion for learning using entity triggers. In the conventional setup for supervised learning for NER, we let x = [ x (1) , x (2) ,    , x ( n ) ] denote a sentence in the labeled training corpus DL . Each labeled sentence has a NER-tag sequence y = [ y (1) , y (2) ,    , y ( n ) ] , where y ( i )  Y and Y can be { O , B-PER ,    } . Thus, we have DL = { ( x i , y i ) } , and an unlabeled corpus DU = { x i } . We propose to annotate entity triggers in sentences. We use T ( x , y ) to represent the set of annotated entity triggers, where each trigger t i  T ( x , y ) is associated with an entity index e and a set of word indices { w i } . Note that we use the index of the first word of an entity as its entity index. That is, t = ( { w 1 , w 2 ,    }  e ) , where e and w i are integers in the range of [1 , | x | ] . Adding triggers creates a new form of data DT = { ( x i , y i , T ( x i , y i ) } . Our goal is to learn a model for NER from a trigger-labeled dataset DT , such that we can achieve comparable learning performance to a model with a much larger DL . 3 Trigger Matching Networks We propose a straightforward yet effective framework, named Trigger Matching Networks (TMN), consisting of a trigger encoder ( TrigEncoder ), a semantic-based trigger matching module ( TrigMatcher ), and a base sequence tagger ( SeqTagger ). We have two learning stages for the framework: the first stage (Section 3.1) jointly learns the TrigEncoder and TrigMatcher , and the second stage (Section 3.2) uses the trigger vectors to learn NER tag labels. 3.1 Trigger Encoding & Semantic Matching Learning trigger representations and semantically matching them with sentences are inseparable tasks. Desired trigger vectors capture the semantics in a shared embedding space with token hidden states, such that sentences and triggers can be semantically matched. Learning an attention-based matching module between entity triggers and sentences is necessary so that triggers and sentences can be semantically matched. Specifically, for a sentence x with multiple entities { e 1 , e 2 ,    } , for each entity e i we assume that there is a set of triggers T i = { t ( i ) 1 , t ( i ) 2 ,    } without loss of generality. To enable more efficient Bidirectional LSTM Networks Sent. Rep. Trigger Rep. Bidirectional LSTM Networks B-PERO O O O O O Trigger Classification Loss CRF  : Learning for Sequence Tagging Contrastive Loss LSM <latexit sha1_base64="5Dr2frp6Y78DxAGDPOmRUM5R8yI=">AAAB7XicbVBNSwMxEJ2tX7V+VT16CRbBU9mVFj0WvXhQqGg/oF1KNs22sdlkSbJCWfofvHhQxKv/x5v/xrTdg7Y+GHi8N8PMvCDmTBvX/XZyK6tr6xv5zcLW9s7uXnH/oKlloghtEMmlagdYU84EbRhmOG3HiuIo4LQVjK6mfuuJKs2keDDjmPoRHggWMoKNlZo3vfT+dtIrltyyOwNaJl5GSpCh3it+dfuSJBEVhnCsdcdzY+OnWBlGOJ0UuommMSYjPKAdSwWOqPbT2bUTdGKVPgqlsiUMmqm/J1IcaT2OAtsZYTPUi95U/M/rJCa88FMm4sRQQeaLwoQjI9H0ddRnihLDx5Zgopi9FZEhVpgYG1DBhuAtvrxMmmdlr1Ku3lVKtcssjjwcwTGcggfnUINrqEMDCDzCM7zCmyOdF+fd+Zi35pxs5hD+wPn8AV6djwE=</latexit> Structured Self-Attention Layer g s <latexit sha1_base64="vYcSCMfjd5sWvKTiT9fKRqAeqCk=">AAAB83icbVDLSsNAFL2pr1pfVZduBovgqiSi6LLoxmUF+4CmlMl00g6dTMLMjVBCf8ONC0Xc+jPu/BsnbRbaemDgcM693DMnSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wSTu9zvPHFtRKwecZrwfkRHSoSCUbSS70cUx0GYjQZmNqjW3Lo7B1klXkFqUKA5qH75w5ilEVfIJDWm57kJ9jOqUTDJZxU/NTyhbEJHvGepohE3/WyeeUbOrDIkYaztU0jm6u+NjEbGTKPATuYZzbKXi/95vRTDm34mVJIiV2xxKEwlwZjkBZCh0JyhnFpCmRY2K2FjqilDW1PFluAtf3mVtC/q3mX96uGy1rgt6ijDCZzCOXhwDQ24hya0gEECz/AKb07qvDjvzsditOQUO8fwB87nD3bhkfk=</latexit> g t <latexit sha1_base64="ESRGTsfbvn7G7hfgLSAZQyYApbU=">AAAB83icbVDLSsNAFL2pr1pfVZduBovgqiSi6LLoxmUF+4CmlMl00g6dTMLMjVBCf8ONC0Xc+jPu/BsnbRbaemDgcM693DMnSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wSTu9zvPHFtRKwecZrwfkRHSoSCUbSS70cUx0GYjQY4G1Rrbt2dg6wSryA1KNAcVL/8YczSiCtkkhrT89wE+xnVKJjks4qfGp5QNqEj3rNU0YibfjbPPCNnVhmSMNb2KSRz9fdGRiNjplFgJ/OMZtnLxf+8XorhTT8TKkmRK7Y4FKaSYEzyAshQaM5QTi2hTAublbAx1ZShraliS/CWv7xK2hd177J+9XBZa9wWdZThBE7hHDy4hgbcQxNawCCBZ3iFNyd1Xpx352MxWnKKnWP4A+fzB3hmkfo=</latexit> type(e) = PERLTC <latexit sha1_base64="ruNI31krsrYxeKepsIhlzZUfJuU=">AAAB7XicbVA9SwNBEJ2LXzF+RS1tFoNgFe4kQctgGguLCPmC5Ah7m71kzd7usbsnhCP/wcZCEVv/j53/xk1yhSY+GHi8N8PMvCDmTBvX/XZyG5tb2zv53cLe/sHhUfH4pK1loghtEcml6gZYU84EbRlmOO3GiuIo4LQTTOpzv/NElWZSNM00pn6ER4KFjGBjpfb9IG3WZ4NiyS27C6B14mWkBBkag+JXfyhJElFhCMda9zw3Nn6KlWGE01mhn2gaYzLBI9qzVOCIaj9dXDtDF1YZolAqW8Kghfp7IsWR1tMosJ0RNmO96s3F/7xeYsIbP2UiTgwVZLkoTDgyEs1fR0OmKDF8agkmitlbERljhYmxARVsCN7qy+ukfVX2KuXqQ6VUu83iyMMZnMMleHANNbiDBrSAwCM8wyu8OdJ5cd6dj2VrzslmTuEPnM8fUPGO+A==</latexit> L <latexit sha1_base64="TOO3Ikl/IgY7t8Ss6ba6sY3RmME=">AAAB6HicbVA9SwNBEJ2LXzF+RS1tFoNgFe4komXQxsIiAfMByRH2NnPJmr29Y3dPCCG/wMZCEVt/kp3/xk1yhSY+GHi8N8PMvCARXBvX/XZya+sbm1v57cLO7t7+QfHwqKnjVDFssFjEqh1QjYJLbBhuBLYThTQKBLaC0e3Mbz2h0jyWD2acoB/RgeQhZ9RYqX7fK5bcsjsHWSVeRkqQodYrfnX7MUsjlIYJqnXHcxPjT6gynAmcFrqpxoSyER1gx1JJI9T+ZH7olJxZpU/CWNmShszV3xMTGmk9jgLbGVEz1MveTPzP66QmvPYnXCapQckWi8JUEBOT2dekzxUyI8aWUKa4vZWwIVWUGZtNwYbgLb+8SpoXZa9SvqxXStWbLI48nMApnIMHV1CFO6hBAxggPMMrvDmPzovz7nwsWnNONnMMf+B8/gCmIYzY</latexit>  : Jointly Training TrigEncoder & TrigMatcher c o n c a t  g t <latexit sha1_base64="v1d7lGKquw2hk4tFpqJWsOrK2hw=">AAAB+3icbVDLSsNAFJ3UV62vWJduBovgqiSi6LLoxmUF+4AmlMl00g6dTMLMjVhCfsWNC0Xc+iPu/BsnbRbaemDgcM693DMnSATX4DjfVmVtfWNzq7pd29nd2z+wD+tdHaeKsg6NRaz6AdFMcMk6wEGwfqIYiQLBesH0tvB7j0xpHssHmCXMj8hY8pBTAkYa2nVvQiDzIgKTIMzGQ8jzod1wms4ceJW4JWmgEu2h/eWNYppGTAIVROuB6yTgZ0QBp4LlNS/VLCF0SsZsYKgkEdN+Ns+e41OjjHAYK/Mk4Ln6eyMjkdazKDCTRUi97BXif94ghfDaz7hMUmCSLg6FqcAQ46IIPOKKURAzQwhV3GTFdEIUoWDqqpkS3OUvr5LuedO9aF7eXzRaN2UdVXSMTtAZctEVaqE71EYdRNETekav6M3KrRfr3fpYjFascucI/YH1+QPWY5T4</latexit> Kzch is the leader of our group Kzch is the leader of our group Bidirectional LSTM Networks Sent. Rep. Bidirectional LSTM Networks Structured Self-Attention Layer g s <latexit sha1_base64="vYcSCMfjd5sWvKTiT9fKRqAeqCk=">AAAB83icbVDLSsNAFL2pr1pfVZduBovgqiSi6LLoxmUF+4CmlMl00g6dTMLMjVBCf8ONC0Xc+jPu/BsnbRbaemDgcM693DMnSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wSTu9zvPHFtRKwecZrwfkRHSoSCUbSS70cUx0GYjQZmNqjW3Lo7B1klXkFqUKA5qH75w5ilEVfIJDWm57kJ9jOqUTDJZxU/NTyhbEJHvGepohE3/WyeeUbOrDIkYaztU0jm6u+NjEbGTKPATuYZzbKXi/95vRTDm34mVJIiV2xxKEwlwZjkBZCh0JyhnFpCmRY2K2FjqilDW1PFluAtf3mVtC/q3mX96uGy1rgt6ijDCZzCOXhwDQ24hya0gEECz/AKb07qvDjvzsditOQUO8fwB87nD3bhkfk=</latexit> c o n c a t  g t <latexit sha1_base64="v1d7lGKquw2hk4tFpqJWsOrK2hw=">AAAB+3icbVDLSsNAFJ3UV62vWJduBovgqiSi6LLoxmUF+4AmlMl00g6dTMLMjVhCfsWNC0Xc+iPu/BsnbRbaemDgcM693DMnSATX4DjfVmVtfWNzq7pd29nd2z+wD+tdHaeKsg6NRaz6AdFMcMk6wEGwfqIYiQLBesH0tvB7j0xpHssHmCXMj8hY8pBTAkYa2nVvQiDzIgKTIMzGQ8jzod1wms4ceJW4JWmgEu2h/eWNYppGTAIVROuB6yTgZ0QBp4LlNS/VLCF0SsZsYKgkEdN+Ns+e41OjjHAYK/Mk4Ln6eyMjkdazKDCTRUi97BXif94ghfDaz7hMUmCSLg6FqcAQ46IIPOKKURAzQwhV3GTFdEIUoWDqqpkS3OUvr5LuedO9aF7eXzRaN2UdVXSMTtAZctEVaqE71EYdRNETekav6M3KrRfr3fpYjFascucI/YH1+QPWY5T4</latexit> LearnedTrigger Rep.Table Mean pooling g t 2 <latexit sha1_base64="yHTqdZezy1IzDpthV/l93GUJabI=">AAAB+3icbVBNS8NAFNzUr1q/Yj16WSyCp5KUih4LXjxWsK3QhrDZbtqlm03YfRFLyF/x4kERr/4Rb/4bN20O2jqwMMy8x5udIBFcg+N8W5WNza3tnepubW//4PDIPq73dZwqyno0FrF6CIhmgkvWAw6CPSSKkSgQbBDMbgp/8MiU5rG8h3nCvIhMJA85JWAk365no4jANAiziZ+B38rz3LcbTtNZAK8TtyQNVKLr21+jcUzTiEmggmg9dJ0EvIwo4FSwvDZKNUsInZEJGxoqScS0ly2y5/jcKGMcxso8CXih/t7ISKT1PArMZBFUr3qF+J83TCG89jIukxSYpMtDYSowxLgoAo+5YhTE3BBCFTdZMZ0SRSiYumqmBHf1y+uk32q67eblXbvRaZZ1VNEpOkMXyEVXqINuURf1EEVP6Bm9ojcrt16sd+tjOVqxyp0T9AfW5w+47pTU</latexit> g t 3 <latexit sha1_base64="MqmPx0Wk6XhalfjWxOfNBIun31g=">AAAB+3icbVBNS8NAFNzUr1q/Yj16WSyCp5BoRY8FLx4r2FpoQ9hsN+3SzSbsvogl5K948aCIV/+IN/+N2zYHbR1YGGbe481OmAquwXW/rcra+sbmVnW7trO7t39gH9a7OskUZR2aiET1QqKZ4JJ1gINgvVQxEoeCPYSTm5n/8MiU5om8h2nK/JiMJI84JWCkwK7ng5jAOIzyUZBDcFEURWA3XMedA68SryQNVKId2F+DYUKzmEmggmjd99wU/Jwo4FSwojbINEsJnZAR6xsqScy0n8+zF/jUKEMcJco8CXiu/t7ISaz1NA7N5CyoXvZm4n9eP4Po2s+5TDNgki4ORZnAkOBZEXjIFaMgpoYQqrjJiumYKELB1FUzJXjLX14l3XPHazqXd81GyynrqKJjdILOkIeuUAvdojbqIIqe0DN6RW9WYb1Y79bHYrRilTtH6A+szx+6dZTV</latexit> g t 4 <latexit sha1_base64="AhhJDMVyfDqfzXnM+na4G9HdGCw=">AAAB+3icbVDLSsNAFJ3UV62vWJduBovgqiRS0WXBjcsK9gFtCJPppB06mYSZG7GE/IobF4q49Ufc+TdO2iy09cDA4Zx7uWdOkAiuwXG+rcrG5tb2TnW3trd/cHhkH9d7Ok4VZV0ai1gNAqKZ4JJ1gYNgg0QxEgWC9YPZbeH3H5nSPJYPME+YF5GJ5CGnBIzk2/VsFBGYBmE28TPwW3me+3bDaToL4HXilqSBSnR8+2s0jmkaMQlUEK2HrpOAlxEFnAqW10apZgmhMzJhQ0MliZj2skX2HJ8bZYzDWJknAS/U3xsZibSeR4GZLILqVa8Q//OGKYQ3XsZlkgKTdHkoTAWGGBdF4DFXjIKYG0Ko4iYrplOiCAVTV82U4K5+eZ30Lptuq3l132q0m2UdVXSKztAFctE1aqM71EFdRNETekav6M3KrRfr3fpYjlascucE/YH1+QO7/JTW</latexit> g t 5 <latexit sha1_base64="YsBPtT4PycpgeQJC/aZTlWrJr0Y=">AAAB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUnEosuCG5cV7APaECbTSTt0MgkzE7GE/IobF4q49Ufc+TdO2iy09cDA4Zx7uWdOkHCmtON8W5WNza3tnepubW//4PDIPq73VJxKQrsk5rEcBFhRzgTtaqY5HSSS4ijgtB/Mbgu//0ilYrF40POEehGeCBYygrWRfLuejSKsp0GYTfxM+608z3274TSdBdA6cUvSgBId3/4ajWOSRlRowrFSQ9dJtJdhqRnhNK+NUkUTTGZ4QoeGChxR5WWL7Dk6N8oYhbE0T2i0UH9vZDhSah4FZrIIqla9QvzPG6Y6vPEyJpJUU0GWh8KUIx2jogg0ZpISzeeGYCKZyYrIFEtMtKmrZkpwV7+8TnqXTfeq2bq/arSbZR1VOIUzuAAXrqENd9CBLhB4gmd4hTcrt16sd+tjOVqxyp0T+APr8we9g5TX</latexit> g t 1 <latexit sha1_base64="NEvlkA3NWHbVsKhnWcog2EZu+q0=">AAAB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUikosuCG5cV7APaECbTSTt0MgkzE7GE/IobF4q49Ufc+TdO2yy09cDA4Zx7uWdOmHKmtOt+W5WNza3tnepubW//4PDIPq53VZJJQjsk4Ynsh1hRzgTtaKY57aeS4jjktBdOb+d+75FKxRLxoGcp9WM8FixiBGsjBXY9H8ZYT8IoHwe5DryiKAK74TruAmideCVpQIl2YH8NRwnJYio04Vipgeem2s+x1IxwWtSGmaIpJlM8pgNDBY6p8vNF9gKdG2WEokSaJzRaqL83chwrNYtDMzkPqla9ufifN8h0dOPnTKSZpoIsD0UZRzpB8yLQiElKNJ8ZgolkJisiEywx0aauminBW/3yOuleOl7TubpvNlpOWUcVTuEMLsCDa2jBHbShAwSe4Ble4c0qrBfr3fpYjlascucE/sD6/AG3Z5TT</latexit> k-nearest leader of  group  PER Pyzxc is the head of our team Pyzxc is the head of our team  : Tagging the Sequence  : Soft-Matching Triggers Trigger-basedGlobalAttention B-PERO O O O O OCRF Trigger-basedGlobalAttention Figure 2: Two-stage training in Trigger Matching Network (Left). batch-based training, we reformat the trigger-based annotated dataset DT such that each new sequence contains only one entity and one trigger. We then create a training instance by pairing each entity with one of its triggers, denoted ( x , e i , t ( i ) j ) . For each reformed training instance ( x , e, t ) , we first apply a bidirectional LSTM (BLSTM) on the sequence of word vectors of x , obtaining a sequence of hidden states that are the contextualized word representations h i for each token x i in the sentence. We use H to denote the matrix containing the hidden vectors of all of the tokens, and we use Z to denote the matrix containing the hidden vectors of all trigger tokens inside the trigger t . In order to learn an attention-based representation of both triggers and sentences, we follow the self-attention method introduced by (Lin et al., 2017b) as follows: (cid:126)a sent = SoftMax (cid:0) W 2 tanh (cid:0) W 1 HT (cid:1)(cid:1) g s = (cid:126)a sent H (cid:126)a trig = SoftMax (cid:0) W 2 tanh (cid:0) W 1 ZT (cid:1)(cid:1) g t = (cid:126)a trig ZW 1 and W 2 are two trainable parameters for computing self-attention score vectors (cid:126)a sent and (cid:126)a trig . We obtain a vector representing the weighted sum of the token vectors in the entire sentence as the final sentence vector g s . Similarly, g t is the final trigger vector, representing the weighted sum of the token vectors in the trigger. We want to use the type of the associated entity as supervision to guide the trigger representation. Thus, the trigger vector g t is further fed into a multi-class classifier to predict the type of the associated entity e (such as PER , LOC , etc) which we use type ( e ) to denote. The loss of the trigger classification is as follows: LTC =  (cid:88) log P ( type ( e ) | g t ;  TC ) , where  TC is a model parameter to learn. Towards learning to match triggers and sentences based on attention-based representations, we use contrastive loss (Hadsell et al., 2006). The intuition is that similar triggers and sentences should have close representations (i.e., have a small distance between them, d ). We create negative examples (i.e., mismatches) for training by randomly mixing the triggers and sentences, because TrigMatcher needs to be trained with both positive and negative examples of the form (sentence, trigger, label). For the negative examples, we expect a margin m between their embed-dings. The contrastive loss of soft matching is as follows, where 1 matched is 1 if the trigger was originally in this sentence and 0 if they are not: d = (cid:107) g s  g t (cid:107) 2 LSM = (1  1 matched )12 ( d ) 2 + 1 matched 1 2 { max (0 , m  d ) } 2 The joint loss of the first stage is thus L = LTC + L SM , where  is a hyper-parameter to tune. The learning objective in this stage is to output the tag sequence y . Following the most common design of neural NER architecture, BLSTM-CRF (Ma and Hovy, 2016), we incorporate the entity triggers as attention queries to train a trigger-enhanced sequence tagger for NER. Note that the BLSTM used in the the TrigEncoder and TrigMatcher modules is the same BLSTM we use in the SeqTagger to obtain H , the matrix containing the hidden vectors of all of the tokens. Given a sentence x , we use the previously trained TrigMatcher to compute the mean of all the trigger vectors  g t associated with this sentence. Following the conventional attention method (Lu-ong et al., 2015), we incorporate the mean trigger vector as the query, creating a sequence of attention-based token representations, H (cid:48) . U 1 , U 2 , and v are trainable parameters for computing the trigger-enhanced attention scores for each token. Finally, we concatenate the original token representation H with the trigger-enhanced one H (cid:48) as the input ( [ H ; H (cid:48) ] ) to the final CRF tagger. Note that in this stage, our learning objective is the same as conventional NER, which is to correctly predict the tag for each token. When inferencing tags on unlabeled sentences, we do not know the sentence's triggers. Instead, we use the TrigMatcher to compute the similarities between the self-attended sentence representations and the trigger representations, using the most suitable triggers as additional inputs to the SeqTagger . Specifically, we have a trigger dictionary from our training data, T = { t | (  ,  , t )  DT } . Recall that we have learned a trigger vector for each of them, and we can load these trigger vectors as a look-up table in memory. For each unlabeled sentence x , we first compute its self-attended vector g s as we do when training the TrigMatcher . Using L2-norm distances to compute the contrastive loss, we efficiently retrieve the most similar triggers in the shared embedding space of the sentence and trigger vectors. Then, we calculate  g t , the mean of the top k nearest semantically matched triggers, as this serves a proxy to triggers mentioned for the entity type in the labeled data. We then use it as the attention query for SeqTagger , similarly in Sec. 3.2. In this section, we first discuss how to collect entity triggers, and empirically study the data-efficiency of our proposed framework. We use a general domain dataset CoNLL2003 (Tjong Kim Sang and De Meul-der, 2003) and a bio-medical domain dataset BC5CDR (Li et al., 2016). Both datasets are well-studied and popular in evaluating the performance of neural named entity recognition models such as BLSTM-CRF (Ma and Hovy, 2016). In order to collect the entity triggers from human annotators, we use Amazon SageMaker Ground Truth 3 to crowd-source entity triggers. More recently, Lee et al. (2020) developed an annotation framework, named LEAN-LIFE, which supports our proposed trigger annotating. Specifically, we sample 20% of each training set as our inputs, and then reform them (Section 2). Annotators are asked to annotate a group of words that would be helpful in typing and/or detecting the occurrence of a particular entity in the sentence. We masked the entity tokens with their types so that human annotators are more focused on the nonentity words in the sentence when considering the triggers. We consolidate multiple triggers for each entity by taking the intersection of the three anno-tators' results. Statistics of the final curated triggers are summarized in Table 1. 4.2 Base model We require a base model to compare with our proposed TMN model in order to validate whether the TMN model effectively uses triggers to improve model performance in a limited label setting. We choose the CNN-BLSTM-CRF (Ma and Hovy, 2016) as our base model for its wide usage in research of neural NER models and applications. Our TMNs are implemented within the same codebase and use the same external word 3 An advanced version of Amazon Mechanical Turk . vectors from GloVE (Pennington et al., 2014). The hyper-parameters of the CNNs, BLSTMs, and CRFs are also the same. This ensures a fair comparison between a typical non-trigger NER model and our trigger-enhanced framework. Labeled data efficiency. We first seek to study the cost-effectiveness of using triggers as an additional source of supervision. Accordingly, we explore the performance of our model and the baseline for different fractions of the training data. The results on the two datasets are shown in Table 2. The full results are shown in Table 3. We can see that by using only 20% of the trigger-annotated data, TMN model delivers comparable performance as the baseline model using 50-70% traditional training data. The drastic improvement in the model performance obtained using triggers thus justifies the slightly additional cost incurred in annotating triggers. Self-training with triggers. We also do a preliminary investigation of adopting self-training (Rosenberg et al., 2005) with triggers. We make inferences on unlabeled data and take the predictions with high confidences as the weak training examples for continually training the model. The confidence is computed following the MNLP metric (Shen et al., 2017), and we take top 20% every epoch. With the self-training method, we further improve the TMN model's F-1 scores by about 0.5  1.0%. Annotation time vs. performance. Although it is hard to accurately study the time cost on the crowd-sourcing platform we use, based on our of-Figure 3: The cost-effectiveness study. fline simulation we argue that annotating both triggers and entities are about 1 . 5 times (BLSTM-CRF (x1.5)) longer than only annotating entities. our offline simulation. In Figure 3, The x-axis for BLSTM-CRF means the number of sentences annotated with only entities, while for TMN means the number of sentences tagged with both entities and triggers. In order to reflect human annotators spending 1.5 to 2 times as long annotating triggers and entities as they spend annotating only entities, we stretch the x-axis for BLSTM-CRF. We can clearly see that the proposed TMN outperforms the BLSTM-CRF model by a large margin. Even if we consider the extreme case that tagging triggers requires twice the human effort (BLSTM-CRF (x2)), the TMN is still significantly more labor-efficient in terms of F1 scores. We introduce entity trigger as a complementary annotation. We crowdsourced triggers on two mainstream datasets and will release them to the community, and proposed a novel framework TMN which can generalize to unseen sentences easily for tagging named entities. This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, NSF SMA 18-29268, and Snap research gift. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government.
abstain abstain abstain abstain method objective abstain abstain abstain abstain objective abstain result method other other objective method objective method method result abstain method method abstain method abstain method abstain other abstain method abstain method method abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective other other