Contextual string embeddings are a recent type of contextualized word embedding that were shown to yield state-of-the-art results when utilized in a range of sequence labeling tasks. They are based on character-level language models which treat text as distributions over characters and are capable of generating embeddings for any string of characters within any textual context. However, such purely character-based approaches struggle to produce meaningful embeddings if a rare string is used in a underspecified context. To address this drawback, we propose a method in which we dynamically aggregate contextualized embeddings of each unique string that we encounter. We then use a pooling operation to distill a global word representation from all contextualized instances. We evaluate these pooled contextualized embeddings on common named entity recognition (NER) tasks such as CoNLL-03 and WNUT and show that our approach significantly improves the state-of-the-art for NER. We make all code and pre-trained models available to the research community for use and reproduction. Word embeddings are a crucial component in many NLP approaches (Mikolov et al., 2013; Pennington et al., 2014) since they capture latent semantics of words and thus allow models to better train and generalize. Recent work has moved away from the original one word, one embed-ding paradigm to investigate contextualized embedding models (Peters et al., 2017, 2018; Akbik et al., 2018). Such approaches produce different embeddings for the same word depending on its context and are thus capable of capturing latent contextualized semantics of ambiguous words. Recently, Akbik et al. (2018) proposed a character-level contextualized embeddings ap-Fung B-PER Permadi E-PER ( Taiwan S-LOC ) v Indra S-ORG Figure 1: Example sentence that provides underspecified context. proach they refer to as contextual string embeddings . They leverage pre-trained character-level language models from which they extract hidden states at the beginning and end character positions of each word to produce embeddings for any string of characters in a sentential context. They showed these embeddings to yield state-of-the-art results when utilized in sequence labeling tasks such as named entity recognition (NER) or part-of-speech (PoS) tagging. Underspecified contexts. However, such contextualized character-level models suffer from an inherent weakness when encountering rare words in an underspecified context. Consider the example text segment shown in Figure 1:  Fung Permadi (Taiwan) v Indra , from the English CO NLL-03 test data split (Tjong Kim Sang and De Meulder, 2003). If we consider the word  Indra  to be rare (meaning no prior occurrence in the corpus used to generate word embeddings), the underspecified context allows this word to be interpreted as either a person or an organization. This leads to an underspecified embedding that ultimately causes an incorrect classification of  Indra  as an organization in a downstream NER task. Pooled Contextual Embeddings. In this paper, we present a simple but effective approach to address this issue. We intuit that entities are normally only used in underspecified contexts if they are expected to be known to the reader. That is, they are either more clearly introduced in an earlier sentence, or part of general in-domain knowl-2 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 NAACL-HLT 2019 Submission ***. Indeed, the string  Indra  in the CO NLL-03 data also occurs in the earlier sentence  Indra Wijaya (Indonesia) beat Ong Ewe Hock . Based on this, we propose an approach in which we dynamically aggregate contextualized embeddings of each unique string that we encounter as we process a dataset. We then call the pooling operation over all contextualized embeddings for this word in the memory (line 4) to compute the pooled contextualized embedding. Finally, we concatenate the original contextual embedding together with the pooled representation, to ensure that both local and global interpretations are represented (line 5). As Algorithm 2 shows, we reset the memory at the beginning of each pass over the training data (line emb context Indra 2 I n d r a W i j a y a b e a t O n g E w e emb context Indra 3 A n d I n d r a s a i d t h a t . . . memory Indra emb proposed F u n g P e r m a d i v I n d r a Character Language Model emb context Indra 1 pooling concatenation current sentence Figure 2: Example of how we generate our proposed embedding ( emb proposed ) for the word  Indra  in the example text segment  Fung Permadi v Indra . We extract a contextual string embedding ( emb context ) for this word and retrieve from the memory all embeddings that were produced for this string on previous sentences. We pool and concatenate all local contextualized embeddings to produce the final embedding. We then use a pooling operation to distill a global word representation from all contextualized instances that we use in combination with the current contextualized representation as new word embedding. edge a reader is expected to have. Indeed, the string  Indra  in the CO NLL-03 data also occurs in the earlier sentence  Indra Wijaya (Indonesia) beat Ong Ewe Hock . We evaluate our proposed embedding approach on the task of named entity recognition on the CO NLL-03 (English, German and Dutch) and WNUT datasets. In all cases, we find that our approach outperforms previous approaches and yields new state-of-the-art scores. We contribute our approach and all pre-trained models to the open source FLAIR 1 framework, to ensure reproducibility of these results. 2 Method Our proposed approach dynamically builds up a memory of contextualized embeddings and applies a pooling operation to distill a global contextualized embedding for each word. Based on this, we propose an approach in which we dynamically aggregate contextualized embeddings of each unique string that we encounter as we process a dataset. We then use a pooling operation to distill a global word representation from all contextualized instances that we use in combination with the current contextualized representation as new word embedding. Our approach thus produces evolving word representations that change over time as more instances of the same word are observed in the data. It requires an embed () function that produces a contextualized embedding for a given word in a sentence context (see Akbik et al. (2018)). It also requires a memory that records for each unique word all previous contextual embeddings, and a pool () operation to pool embedding vectors. We evaluate our proposed embedding approach on the task of named entity recognition on the CO NLL-03 (English, German and Dutch) and WNUT datasets. In all cases, we find that our approach outperforms previous approaches and yields new state-of-the-art scores. We contribute our approach and all pre-trained models to the open source FLAIR 1 framework (Akbik et al., 2019), to ensure reproducibility of these results. It requires an embed () function that produces a contextualized embedding for a given word in a 1 https://github.com/zalandoresearch/flair sentence context (see Akbik et al. (2018)). This means that the resulting pooled contextualized embedding has twice the dimensionality of the original embedding. It also requires a memory that records for each unique word all previous contextual embeddings, and a pool () operation to pool embedding vectors. Algorithm 1 Compute pooled embedding Input: sentence , memory 1: for word in sentence do 2: emb context  embed( word ) within sentence 3: add emb context to memory [ word ] 4: emb pooled  pool( memory [ word ] ) 5: word.embedding  concat( emb pooled , emb context ) 6: end for Crucially, our approach expands the memory each time we embed a word. Therefore, the same word in the same context may have different embeddings over time as the memory is built up. This is illustrated in Algorithm 1: to embed a word (in a sentential context), we first call the embed () function (line 2) and add the resulting embedding to the memory for this word (line 3). We then call the pooling operation over all contextualized embeddings for this word in the memory (line 4) to compute the pooled contextualized embedding. Finally, we concatenate the original contextual embedding together with the pooled representation, to ensure that both local and global interpretations are represented (line 5). This means that the resulting pooled contextualized embedding has twice the dimensionality of the original embedding. Pooling operations. We experiment with different pooling operations: mean pooling to average a word's contextualized embedding vectors, and min and max pooling to compute a vector of all Approach CO NLL-03 ENCO NLL-03 DECO NLL-03 NL WNUT-17 Pooled Contextualized Embeddings min 93.18  0.09 88.27  0.30 90.12  0.14 49.07  0.31 Pooled Contextualized Embeddings max 93.13  0.09 88.05  0.25 90.26  0.10 49.05  0.26 Pooled Contextualized Embeddings mean 93.10  0.11 87.69  0.27 90.44  0.20 49.59  0.41 Contextual String Emb. Training downstream models. When training downstream task models (such as for NER), we typically make many passes over the training data. As Algorithm 2 shows, we reset the memory at the beginning of each pass over the training data (line 2), so that it is build up from scratch at each epoch. This approach ensures that the downstream task model learns to leverage pooled embeddings that are built up (e.g. evolve ) over time. It also ensures that pooled embeddings during training are only computed over training data. After training, (i.e. during NER prediction), we do not reset embeddings and instead allow our approach to keep expanding the memory and evolve the embeddings. We verify our proposed approach in four named entity recognition (NER) tasks: We use the English, German and Dutch evaluation setups of the CO NLL-03 shared task (Tjong Kim Sang and De Meulder, 2003) to evaluate our approach on classic newswire data, and the WNUT-17 task on emerging entity detection (Derczynski et al., 2017) to evaluate our approach in a noisy user-generated data setting with few repeated entity mentions. We use the open source FLAIR framework in all our experiments. It implements the standard BiLSTM-CRF sequence labeling architecture (Huang et al., 2015) and includes pre-trained contextual string embeddings for many languages. To FLAIR , we add an implementation of our proposed pooled contextualized embeddings . Hyperparameters. For our experiments, we follow the training and evaluation procedure outlined in Akbik et al. (2018) and follow most hyperpa-rameter suggestions as given by the in-depth study presented in Reimers and Gurevych (2017). That is, we use an LSTM with 256 hidden states and one layer (Hochreiter and Schmidhuber, 1997), a locked dropout value of 0.5, a word dropout of 0.05, and train using SGD with an annealing rate of 0.5 and a patience of 3. We perform model selection over the learning rate  { 0 . 01 , 0 . 05 , 0 . 1 } and mini-batch size  { 8 , 16 , 32 } , choosing the model with the best F -measure on the validation set. Following Peters et al. (2017), we then repeat the experiment 5 times with different random seeds, and train using both train and development set, reporting both average performance and standard deviation over these runs on the test set as final performance. Standard word embeddings. The default setup of Akbik et al. (2018) recommends contextual string embeddings to be used in combination with standard word embeddings. We use GLOVE embeddings (Pennington et al., 2014) for the English tasks and FASTTEXT embeddings (Bojanowski et al., 2017) for all newswire tasks. Baselines. Our baseline are contextual string embeddings without pooling, i.e. the original setup proposed in Akbik et al. (2018) 2 . By comparing against this baseline, we isolate the impact of our proposed pooled contextualized embeddings. 2 Our reproduced numbers are slightly lower than we reported in Akbik et al. (2018) where we used the official CO NLL-03 evaluation script over BILOES tagged entities. This introduced errors since this script was not designed for S-tagged entities. In addition, we list the best reported numbers for the four tasks. This includes the recent BERT approach using bidirectional transformers by Devlin et al. (2018), the semi-supervised multitask learning approach by Clark et al. (2018), the ELMo word-level language modeling approach by Peters et al. (2018), and the best published numbers for WNUT-17 (Aguilar et al., 2018) and German and Dutch CO NLL-03 (Lample et al., 2016). New state-of-the-art scores. We find that our approach outperforms all previously published results, raising the state-of-the-art for CO NLL-03 on English to 93.18 F1-score (  0.32 pp vs. previous best), German to 88.27 (  0.86 pp) and Dutch to 90.44 (  0.28 pp). The consistent improvements against the contextual string embeddings baseline indicate that our approach is generally a viable option for embedding entities in sequence labeling. Less pronounced impact on WNUT-17. However, we also find no significant improvements on the WNUT-17 task on emerging entities. Depending on the pooling operation, we find comparable results to the baseline. This result is expected since most entities appear only few times in this dataset, giving our approach little evidence to aggregate and pool. Nevertheless, since recent work has not yet experimented with contextual embeddings on WNUT, as side result we report a new state-of-the-art of 49.59 F1 vs. the previous best reported number of 45.55 (Aguilar et al., 2018). Pooling operations. Comparing the pooling operations discussed in Section 2, we generally find similar results. As Table 1 shows, min pooling performs best for English and German CoNLL, while mean pooling is best for Dutch and WNUT. To better isolate the impact of our proposed approach, we run experiments in which we do not use any classic word embeddings, but rather rely solely on contextual string embeddings. As Table 2 shows, we observe more pronounced improvements of pooling vis-a-vis the baseline approach in this setup. This indicates that pooled contextualized embeddings capture global semantics words similar in nature to classical word embeddings. We presented a simple but effective approach that addresses the problem of embedding rare strings in underspecified contexts. Our experimental evaluation shows that this approach improves the state-of-the-art across named entity recognition tasks, enabling us to report new state-of-the-art scores for CO NLL-03 NER and WNUT emerging entity detection. These results indicate that our embedding approach is well suited for NER. Evolving embeddings. Our dynamic aggregation approach means that embeddings for the same words will change over time, even when used in exactly the same contexts. Assuming that entity names are more often used in well-specified contexts, their pooled embeddings will improve as more data is processed. The embedding model thus continues to learn from data even after the training of the downstream NER model is complete and it is used in prediction mode. We consider this idea of constantly evolving representations a very promising research direction. Future work. Our pooling operation makes the conceptual simplification that all previous instances of a word are equally important. However, we may find more recent mentions of a word such as words within the same document or news cycle to be more important for creating embeddings than mentions that belong to other documents or news cycles. Future work will therefore examine methods to learn weighted poolings of previous mentions. We will also investigate applicability of our proposed embeddings to tasks beside NER. Public release. We contribute our code to the FLAIR framework 3 . This allows full reproduction of all experiments presented in this paper, and al-3 The proposed embedding is added to FLAIR in release 0.4.1. as the PooledFlairEmbeddings class (see Akbik et al. (2019) for more details). We would like to thank the anonymous reviewers for their helpful comments. This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement no 732328 (FashionBrain).
abstain abstain abstain objective method result method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method other abstain objective method method other method method objective abstain abstain objective objective objective objective objective objective result abstain abstain objective objective abstain other abstain abstain method abstain method method method abstain abstain method abstain method abstain abstain abstain abstain other method abstain method abstain method method method abstain abstain abstain method abstain abstain method abstain method method method abstain method other abstain result abstain abstain result result result abstain abstain result abstain abstain result abstain method objective result abstain method abstain abstain method abstain abstain result abstain objective abstain objective objective abstain other other