What is the performance of large state-of-the-art models on these datasets?
Akbik et al. (2019) -  89.3 on Ontonotes 5
Baevski et al. (2019) 93.5 on CoNLL-03
Recent attempts to ingest external knowledge into neural models for named-entity recognition (NER) have exhibited mixed results. In this work, we present GazSelfAttn, a novel gazetteer embedding approach that uses self-attention and match span encoding to build enhanced gazetteer embeddings. In addition, we demonstrate how to build gazetteer resources from the open source Wikidata knowledge base. Evaluations on CoNLL-03 and Ontonotes 5 datasets, show F1 improvements over baseline model from 92.34 to 92.86 and 89.11 to 89.32 respectively, achieving performance comparable to large state-of-the-art models.
Named-entity recognition (NER) is the task of tagging relevant entities such as person, location and organization in unstructured text. Modern NER has been dominated by neural models BIBREF0, BIBREF1 combined with contextual embeddings from language models (LMs) BIBREF2, BIBREF3, BIBREF4. The LMs are pre-trained on large amounts of unlabeled text which allows the NER model to use the syntactic and semantic information captured by the LM embeddings. On the popular benchmark datasets CoNLL-03 BIBREF5 and Ontonotes 5 BIBREF6, neural models with LMs achieved state-of-the-art results without gazetteers features, unlike earlier approaches that heavily relied on them BIBREF7. Gazetteers are lists that contain entities such as cities, countries, and person names. The gazetteers are matched against unstructured text to provide additional features to the model. Data for building gazetteers is available for multiple language from structured data resources such as Wikipedia, DBpedia BIBREF8 and Wikidata BIBREF9. In this paper, we propose GazSelfAttn, a novel gazetteer embedding approach that uses self-attention and match span encoding to build enhanced gazetteer representation. GazSelfAttn embeddings are concatenated with the input to a LSTM BIBREF10 or CNN BIBREF11 sequence layer and are trained end-to-end with the model. In addition, we show how to extract general gazetteers from the Wikidata, a structured knowledge-base which is part of the Wikipedia project. Our contributions are the following: [topsep=1pt, leftmargin=15pt, itemsep=-1pt] We propose novel gazetteer embeddings that use self-attention combined with match span encoding. We enhance gazetteer matching with multi-token and single-token matches in the same representation. We demonstrate how to use Wikidata with entity popularity filtering as a resource for building gazetteers. GazSelfAttn evaluations on CoNLL-03 and Ontonotes 5 datasets show F$_1$ score improvement over baseline model from 92.34 to 92.86 and from 89.11 to 89.32 respectively. Moreover, we perform ablation experiments to study the contribution of the different model components.