There is an increasing interest in studying natural language and computer code together, as large corpora of programming texts become readily available on the Internet. For example, StackOverflow currently has over 15 million programming related questions written by 8.5 million users. Meanwhile, there is still a lack of fundamental NLP techniques for identifying code tokens or software-related named entities that appear within natural language sentences. In this paper, we introduce a new named entity recognition (NER) corpus for the computer programming domain, consisting of 15,372 sentences annotated with 20 fine-grained entity types. We trained in-domain BERT representations (BERTOver-flow) on 152 million sentences from StackOverflow, which lead to an absolute increase of +10 F 1 score over off-the-shelf BERT. We also present the SoftNER model which achieves an overall 79.10 F 1 score for code and named entity recognition on StackOverflow data. Our SoftNER model incorporates a context-independent code token classifier with corpus-level features to improve the BERT-based tagging model. 1 1 Introduction Recently there has been significant interest in modeling human language together with computer code (Quirk et al., 2015; Iyer et al., 2016; Yin and Neubig, 2018), as more data becomes available on websites such as StackOverflow and GitHub. This is an ambitious yet promising direction for scaling up language understanding to richer domains. Access to domain-specific NLP tools could help a wide range of downstream applications. For example, extracting software knowledge bases from 1 Our code and data are available at: https:// github.com/jeniyat/StackOverflowNER/ Figure 1: Examples of software-related named entities in a StackOverflow post. text (Movshovitz-Attias and Cohen, 2015), developing better quality measurements of StackOverflow posts (Ravi et al., 2014), finding similar questions (Amirreza Shirani, 2019) and more. However, there is a lack of NLP resources and techniques for identifying software-related named entities (e.g., variable names or application names) within natural language texts. In this paper, we present a comprehensive study that investigates the unique challenges of named entity recognition in the social computer programming domain. These named entities are often ambiguous and have implicit reliance on the accompanied code snippets. For example, the word  list ' commonly refers to a data structure, but can also be used as a variable name (Figure 1). In order to recognize these entities, we propose a software-related named entity recognizer (Soft-NER) that utilizes an attention network to combine the local sentence-level context with corpus-level information extracted from the code snippets. Using our newly annotated corpus of 15,372 sentences in StackOverflow, we rigorously test our proposed SoftNER model, which outperforms BiLSTM-CRF model and fine-tuned BERT model for identifying 20 types of software-related named entities. Our key contributions are the following:  A new StackOverflow NER corpus manually annotated with 20 types of named entities, including all in-line code within natural language sentences (2). We demonstrate that NER in the software domain is an ideal benchmark task for testing effectiveness of contextual word representations, such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019), due to its inherent polysemy and salient reliance on context. An in-domain trained neural SoftNER tagger for StackOveflow (3) that can recognize 20 fine-grained named entity types related to software developing. We also tested its performance on GitHub data of readme files and issue reports. A code token recognizer (3.1) that utilizes StackOveflow code snippets to capture the spelling patterns of code-related tokens, and consistently improves the NER tagger. In-domain pretrained ELMo and BERT representations (3.3) on 152 million sentences from StackOverflow that significantly outperforms off-the-shelf ELMo and leads to more than 21 points increase in F 1 score over off-the-shelf BERT. Overall, our named entity tagger (SoftNER) achieves a 79.10% F 1 score on StackOverflow and 61.08% F 1 score on GitHub data for extracting the 20 software related named entity types. We believe this performance is sufficiently strong to be practically useful. We have released our data and code, including the named entity tagger, our annotated corpus, annotation guideline, a specially designed tokenizer, and pre-trained StackOverflow BERT and ELMo embeddings. In this section, we describe the construction of our StackOverflow NER corpus. We randomly selected 1,237 question-answer threads from StackOverflow 10-year archive (from September 2008 to March 2018) and manually annotated them with 20 types of entities. For each question, four answers were annotated, including the accepted answer, the most upvoted answer, as well as two randomly selected answers (if they exist). Table 1 shows the statistics of our corpus. 40 % of the question-answer threads were double-annotated, which are used as the development and test sets in our experiments (4). We also annotated 6,501 sentences from GitHub readme files and issue reports as additional evaluation data. We defined and annotated 20 types of fine-grained entities, including 8 code-related entities and 12 natural language entities. The code entities include mentions of CLASS , VARIABLE , INLINECODE , FUNCTION , LIBRARY , VALUE , DATATYPE , and HTML XML TAG . Whereas the natural language entities include mentions of APPLICATION , UI ELEMENT , LANGUAGE , DATASTRUCTURE , ALGORITHM , FILETYPE , FILENAME , VERSION , DEVICE , OS, WEBSITE , and USERNAME . Our annotation guideline was developed through several pilots and further updated with notes to resolve difficult cases as the annotation progressed. 2 Each entity type was defined to encourage maximum span length (e.g.,  SGML parser ' instead of  SGML '). We annotated noun phrases without including modifiers (e.g.,  C ' instead of  Plain C '), except a few special cases (e.g.,  rich text ' as a common FILETYPE ). On average, an entity contains about 1.5 tokens. While VARIABLE , FUNCTION and CLASS names mostly consist of only a single token, our annotators found that some are written as multiple tokens when mentioned in natural language text (e.g.,  array list ' for  ArrayList ' in Figure 1). The annotators were asked to read relevant code blocks or software repositories to make a decision, if needed. Annotators also searched Google or Wikipedia to categorize unfamiliar cases. The annotators were asked to update, correct, or add annotations from the user provided (cid:104) code (cid:105) markdown tags. StackOverflow users can utilize (cid:104) code (cid:105) markdowns to highlight the code entities 2 Our annotation guideline is available at: https:// github.com/jeniyat/StackOverflowNER/ . within the natural language sentences. However, in reality, many users do not enclose the code snippets within the (cid:104) code (cid:105) tags; and sometimes use them to highlight non-code elements, such as email addresses, user names, or natural language words. While creating the StackOverflow NER corpurs, we found that 59.73% of code-related entities are not marked by the StackOverflow users. Moreover, only 75.54% of the (cid:104) code (cid:105) enclosed texts are actually code-related, while 10.12% used to are highlighting natural language texts. The rest of cases are referring to non-code entities, such as SOFTWARENAMES and VERSIONS . While markdown tag could be a useful feature for entity segmentation (3.1.3), we emphasize the importance of having a human annotated corpus for training and evaluating NLP tools in the software domain. Our corpus was annotated by four annotators who are college students majored in computer science. We used a web-based annotation tool, BRAT (Stenetorp et al., 2012), and provided annotators with links to the original post on StackOverflow. For every iteration, each annotator was given 50 question-answer threads to annotate, 20 of which were double-annotated. An adjudicator then discussed disagreements with annotators, who also cross-checked the 30 single-annotated questions in each batch. The inter-annotator agreement is 0.62 before adjudication, measured by span-level Cohen's Kappa (Cohen, 1960). To better understand the domain adaptability of our work, we further annotated the readme files and issue reports from 143 randomly sampled repositories in the GitHub dump (Gousios and Spinellis, 2012) (from October 29, 2007 to December 31, 2017). We removed all the code blocks from the issue reports and readme files collected from these 143 repositories. The resulting GitHub NER dataset consists of 6,510 sentences and 10,963 entities of 20 types labeled by two in-house annotators. The inter-annotator agreement of this dataset is 0.68, measured by span-level Cohen's Kappa. We designed a new tokenizer, SOTOKENIZER specifically for the social computer programming domain. StackOverflow and GitHub posts exhibit common features of web texts, including abbreviations, emoticons, URLs, ungrammatical sentences and spelling errors. We found that tokenization is non-trivial as many code-related tokens are mistakenly split by the existing web-text tokenizers, including the CMU Twokenizer (Gimpel et al., 2011), Stanford TweetTokenizer (Manning et al., 2014), and NLTK Twitter Tokenizer (Bird et al., 2009): txScope.Complete() [ txScope' .' Complete' (' )' ] std::condition variable [ std' :' :' condition variable' ] math.h [ math' .' h' ] (cid:104) span (cid:105) [  (cid:104) ' span'  (cid:105) ' ] a==b [ a' =' =' b' ] Therefore, we implemented a new tokenizer, using Twokenizer 3 as the starting point and added additional regular expression rules to avoid splitting code-related tokens. The extraction of software-related named entities imposes significant challenges as it requires resolving a significant amount of unseen tokens, inherent polysemy, and salient reliance on context. Unlike news or biomedical data, spelling patterns and long-distance dependencies are more crucial in the software domain to resolve ambiguities and categorize unseen words. Taken in isolation, many tokens are highly ambiguous and can refer to either programming concepts or common English words, such as:  go ',  react ',  spring ',  while ',  if ',  select '. To address these challenges, we design the SoftNER model that leverages sentential context to disambiguate and domain-specific character representations to handle rare words. Figure 2 shows the architecture of our model, which consists of primarily three components:  An input embedding layer (3.1) that extracts contextualized embeddings from the BERT base model and two new domain-specific embeddings for each word in the input sentence. A embedding attention layer (3.2) that combines the three word embeddings using an attention network. A linear-CRF layer that predicts the entity type of each word using the attentive word representations from the previous layer. 3 https://github.com/myleott/ ark-twokenize-py Figure 2: Our SoftNER model. For each word in the input sentence, we extract in-domain BERT (Devlin et al., 2019) representations and two new domain-specific embeddings produced by (i) a Code Recognizer , which represents if a word can be part of a code entity regardless of context; and (ii) an Entity Segmenter , that predicts whether a word is part of any named entity in the given sentence. Each domain-specific embedding is created by passing a binary value, predicted by a network independent from the SoftNER model. We describe the two standalone auxiliary models that generate these domain-based vectors below. Texts in the software engineering domain contain programming language tokens, such as variable names or code segments, interspersed with natural language words. This makes input representations pre-trained on general book or Wikipedia texts unsuitable for software domain. We pre-trained different in-domain word embeddings, including BERT (BERTOverflow), ELMo (ELMoVerflow), and GloVe (GloVerflow) vectors on the StackOverflow 10-year archive 4 of 152 million sentences and 2.3 billion tokens (3.3). Humans with prior programming knowledge can easily recognize that  list() ' is code,  list ' can be either code or a common English word, whereas  listing ' is more likely a non-code natural language token. We thus introduce a code recognition module to capture such prior probability of how 4 https://archive.org/details/ stackexchange likely a word can be a code token without considering any contextual information. It is worth noting that this standalone code recognition model is also useful for language-and-code research, such as retrieving code snippets based on natural language queries (Iyer et al., 2016; Giorgi and Bader, 2018; Yao et al., 2019) Our code recognition model ( Code Recognizer ) is a binary classifier. It utilizes language model features and spelling patterns to predict whether a word is a code entity. The input features include unigram word and 6-gram character probabilities from two language models (LMs) that are trained on the Gigaword corpus (Napoles et al., 2012) and all the code-snippets in the StackOverflow 10-year archive respectively. We also pre-trained FastText (Joulin et al., 2016) word embeddings using these code-snippets, where a word vector is represented as a sum of its character ngrams. We first transform each ngram probability into a k -dimensional vector using Gaussian binning (Maddela and Xu, 2018), which has shown to improve the performance of neural models using numeric features (Sil et al., 2017; Liu et al., 2016; Maddela and Xu, 2018). We then feed the vec-torized features into a linear layer, concatenate the output with FastText character-level embeddings, and pass them through another hidden layer with sigmoid activation. We predict the token as a code-entity if the output probability is greater than 0 . 5 . This binary prediction is then converted into a vector and used as an input to the SoftNER model. The segmentation task refers to identifying entity spans without assigning entity category. Entity segmentation is simpler and less error-prone than entity recognition as it does not require a fine-grained classification of the entity types. In fact, a segmentation model ( Entity Segmenter ) trained on our annotated StackOverflow corpus can achieve 90.41% precision on the dev set (de-tails in 4.5), predicting whether each token is a part of entity in the given sentence. Our segmentation model fine-tunes the in-domain BERT after concatenating it with two hand-crafted features:  Word Frequency represents the word occurrence count in the training set. As many code tokens are defined by individual users, they occur much less frequently than normal English words. In fact, code and non-code tokens have an average frequency of 1.47 and 7.41 respectively in our corpus. Moreover, ambiguous token that can be either code or non-code entities, such as  windows ', have a much higher average frequency of 92.57. To leverage this observation, we include word frequency as a feature, converting the scalar value into a k -dimensional vector by Gaussian binning (Maddela and Xu, 2018). Code Markdown indicates whether the given token appears inside a (cid:104) code (cid:105) markdown tag in the StackOverflow post. It is worth noting that (cid:104) code (cid:105) tags are noisy as users do not always enclose inline code in a (cid:104) code (cid:105) tag or sometimes use the tag to highlight non-code texts (details in 2.1). Nevertheless, we find it helpful to include the markdown information as a feature as it improves the performance of our segmentation model. The inclusion of hand-crafted features is influ-enced by Wu et al. (2018), where word-shapes and POS tags were shown to improve the performance of sequence tagging models. For each input word w i in the input sentence, we have three embeddings: BERT ( w i 1 ), Code Recognizer ( w i 2 ), and Entity Segmenter ( w i 3 ). We introduce the embedding-level attention  it ( t  { 1 , 2 , 3 } ), which captures each embedding's contribution towards the meaning of the word, to combine them together. To compute  it , we pass the input embeddings through a bidirectional GRU and generate their corresponding hidden representations h it =  GRU ( w it ) . These vectors are then passed through a non-linear layer, which outputs u it = tanh ( W e h it + b e ) . We introduce an embedding-level context vector u e , which is randomly initialized and updated during the training process. This context vector is combined with the hidden embedding representation using a softmax function to extract weight of the embeddings:  it = exp ( u itT u e ) (cid:80) t exp ( u itT u e ) . Finally, we create the word vector by a weighted sum of all the information from different embeddings as word i = (cid:80) t  it h it . The aggregated word vector word i is then fed into a linear-CRF layer, which predicts the entity category for each word based the BIO tagging schema. We use PyTorch framework to implement our proposed SoftNER model and its two auxiliary components, namely code recognition and entity segmentation. The input to the SoftNER model include 850-dimensional vectors extracted from both the code recognizer and the entity segmenter. We pre-trained BERT base , ELMo and GloVe vectors on 152 million sentences from the StackOverflow, excluding sentences from the 1,237 posts in our annotated corpus. The pretraining of the 768-dimensional BERT base model with 64,000 WordPiece vocabulary took 7 days on a Google TPU. The pre-training of 1024-dimensional ELMo vectors took 46 days on 3 NVIDIA Titan X Pascal GPUs. The pre-training of 300-dimensional GloVe embeddings (Penning-ton et al., 2014) with a frequency cut-off of 5 took 8 hours on a server with 32 CPU cores and 386 GB memory. We train the SoftNER model and the two auxiliary models separately. Our segmentation model follows the simple BERT fine-tuning architecture except for the input, where BERT embeddings are concatenated with 100-dimensional code markdown and 10-dimensional word frequency features. We set the number of bins k to 10 for Gaussian vectorization. Our code recognition model is a feedforward network with two hidden layers and a single output node with sigmoid activation. In this section, we show that our SoftNER model outperforms all the previous NER approaches on the StackOverflow and GitHub data. We also discuss the factors pivotal to the performance of our model, namely pre-trained in-domain BERT embeddings and two domain-specific auxiliary tasks. We train and evaluate our SoftNER model on the StackOverflow NER corpus of 9,352 train, 2,942 development and 3,115 test sentences we constructed in 2. We use the same data for our segmentation model but replace all the entity tags with an I-ENTITY tag. For the code recognition model, we created a new lexicon of 6,000 unique tokens randomly sampled from the training set of the StackOverflow NER corpus. Each token was labelled independently without context as CODE , AMBIGUOUS or NON-CODE by two annotators with computer science background. The inter-annotator agreement was 0.89, measured by Cohen's Kappa. After discarding disagreements, we divided the remaining 5,312 tokens into 4,312 train and 1,000 test instances. Then, we merged AMBIGUOUS and NON-CODE categories to facilitate binary classification. We name this dataset of 5312 individual tokens as SOLEXICON . We compare our model with the following baseline and state-of-the-art approaches: A Feature-based Linear CRF model which uses the standard orthographic, context and gazetteer features, along with the code markdown tags and handcrafted regular expressions to recognize code entities (details in Appendix A). A BiLSTM-CRF model with in-domain ELMo embeddings ( ELMoVerflow ; details in 3.3). This architecture is used as the state-of-the-art baseline named-entity recognition models in various domains (Lample et al., 2016; Kulkarni et al., 2018; Dai et al., 2019). An Attentive BiLSTM-CRF model with in-domain ELMo embeddings as well as domain-specific embeddings from the code recognizer and the entity segmenter. This model combines these three word embeddings using an attention network and then utilizes a BiLSTM-CRF layer to predict the entity type of each input word (details in Appendix B). A Fine-tuned out-of-domain BERT model where we fine-tune the original BERT base cased checkpoint 5 on our annotated corpus. A Fine-tuned in-domain BERT model where we fine-tune the in-domain pre-trained BERT base ( BERTOverflow ; details in 3.3) cased checkpoint 6 on our annotated corpus. Table 2 shows the precision ( P ), recall ( R ) and F 1 score comparison of different models evaluated on the StackOverflow NER corpus. Our SoftNER model outperforms the existing NER approaches in all the three metrics. Fine-tuning over in-domain trained BERT (BERTOverflow), in particular, improves F 1 score by more than 10 points in comparison to using the original BERT. Table 3 shows the performance comparison between in-domain and out-of-domain word embeddings. We consider off-the-shelf BERT (De-vlin et al., 2019), ELMo (Peters et al., 2018) and GloVe (Pennington et al., 2014) vectors trained on newswire and web texts as out-of-domain embeddings. When using the BiLSTM-CRF model (Lample et al., 2016; Kulkarni et al., 2018; Dai et al., 2019), we observe a large increase of 13.64 F 1 score when employing in-domain ELMo (ELMoVerflow) representations over in-domain GloVe (GloVeOverflow), and an increase of 15.71 F 1 score over out-of-domain ELMo. We found that fine-tuning out-of-domain BERT (De-vlin et al., 2019) outperforms the out-of-domain 5 https://github.com/google-research/ BERT 6 https://github.com/lanwuwei/ BERTOverflow/ P R F 1 out-of-domain Word Embeddings GloVe (newswire+Wiki+Web) 61.71 49.08 54.67 ELMo (newswire+Wiki) 67.66 47.41 55.75 Fine-tuned BERT (book+Wiki) 45.92 77.02 57.54 In-Domain Word Embeddings GloVeOverflow 66.28 51.28 57.82 ELMoVerflow 74.44 68.71 71.46 Fine-tuned BERTOverflow 72.11 70.51 71.30 Table 3: Performance of fine-tuned BERT model, BiLSTM-CRF model with GloVe and ELMo embeddings on the dev set of our StackOverflow NER corpus. ELMo (Table 3), although it underperforms in-domain ELMo (ELMoVerflow) by 12.92 F 1 score and in-domain BERt (BERTOverflow) by 12.76 F 1 score (Table 2). Similarly, in-domain ELMo outperforms the out-of-domain fine-tuned BERT by 10.67 F 1 score on Github data (Table 8; more details in 5). It is worth noting that, the performance improvements from contextual word embeddings are more pronounced on our software domain than on newswire and biomedical domains. Original ELMo and BERT outperform GloVe by 2.06 and 2.12 points in F 1 respectively on CoNLL 2003 NER task of newswire data (Peters et al., 2018; Devlin et al., 2019). For biomedical domain, in-domain ELMo outperforms out-of-domain ELMo by only 1.33 points in F 1 on the BC2GM dataset (Sheikhshabbafghi et al., 2018). We hypothesized that the performance gains from the in-domain contextual embeddings are largely aided by the model's ability to handle ambiguous and unseen tokens. The increase in performance is especially notable (41%  70% accuracy) for unseen tokens, which constitute 38% of the tokens inside gold entity spans in our dataset. This experiment also demonstrates that our annotated NER corpus provides an attractive test-bed for measuring the adaptability of different contextual word representations. The domain-specific vectors produced by the Code Recognizer and the Entity Segmenter are also crucial for the overall performance of our SoftNER model. Table 4 shows an ablation study. Removing code recognizer vectors and entity segmenter vectors results in a drop of 2.19 and 3.69 in F 1 scores respectively. If we replace embedding-level attention with a simple concatenation of em-P R F 1 SoftNER 78.81 81.72 80.24  Embedding Attention 75.83 79.09 77.43  Code Recognizer 78.76 77.35 78.05  Entity Segmenter 77.82 75.32 76.55 Table 4: Ablation study of SoftNER on the dev set of StackOverflow NER corpus. FastText Embeddings 76.12 81.69 78.81 Table 5: Evaluation results and feature ablation of our code recognition model on SOLEXICON test set of 1000 manually labeled unique tokens, which are sampled from the train set of StackOverflow NER corpus. beddings, the performance also drop by 2.81 F 1 . In addition, we evaluate the effectiveness of our two domain-specific auxiliary systems on their respective tasks. Code Recognition: Table 5 compares the performance of our code recognition model with other baselines on the SLEXICON test set (4.1), which consists of 1,000 random words from the train set of StackOverflow NER corpus classified as either a code or a non-code token. The baselines include: (i) a Most Frequent Label baseline, which assigns the most frequent label according to the human annotation in SOLEXICON train set; and (ii) a frequency baseline, which learns a threshold over token frequency in the train set of StackOverflow NER corpus using a decision tree classifier. Our model outperforms both baselines in terms of F 1 score. Although the most frequent label baseline achieves better precision than our model, it performs poorly on unseen tokens resulting in a large drop in recall and F 1 score. The ablation experiments show that the FastText word embeddings along with the character and word-level features are crucial for the code recognition model. Entity Segmentation: Table 6 shows the performance of our segmentation model on the dev set of our StackOverflow corpus, where the entity tags are replaced by an I-ENTITY tag. Our model achieves an F 1 score of 88.09 and with 90.41% precision and 85.89% recall. Incorporating word frequency and code markdown feature increases the F 1 score by 1.57 and 2.66 points respectively. The low 10.5 F 1 score of Stanford NERP R F 1 Stanford NER Tagger 63.02 5.74 10.52 Our Entity Segmentation Model 90.41 85.89 88.09  Word Frequency 88.32 84.79 86.52  Code Markdown 86.23 84.64 85.43 Table 6: Evaluation of our segmentation model on the dev set of the StackOverflow NER corpus. tagger (Manning et al., 2014), which is trained on newswire text, demonstrates the importance of domain-specific tools for the software engineering domain. Based on our manual inspection, the incorrect predictions made by NER systems on StackOverflow data can be largely classified into the following two categories (see examples in Table 7): Segmentation Mismatch refers to the cases where model predicts the boundary of entities incorrectly. Our SoftNER model reduces such segmentation errors by 89.36% compared to the fine-tuned BERTOverflow baseline. Entity-Type Mismatch refers to the errors where a code entity (e.g., names of variables) is predicted as a non-code entity (e.g., names of devices), and vice-versa. Our SoftNER model reduces such entity type errors by 13.54% compared to the fine-tuned BERTOverflow baseline. As illustrated in Figure 3, our SoftNER model reduced the errors in both categories by incorporating the auxiliary outputs from segmenter and code recognizer model. To understand the domain adaptability of our StackOverflow based SoftNER, we evaluate its performance on readme files and issue reports from 143 randomly sampled repositories in the GitHub dump (Gousios and Spinellis, 2012). We also trained ELMo embeddings (ELMoGithub) on 4 million sentences from randomly sampled 5,000 GitHub repositories. Table 8 shows that the performance of our SoftNER model using StackOverflow ELMo embeddings is similar to the top performing BiLSTM-CRF model using GitHub ELMo embeddings with a difference of only 1.61 points in F 1 . We also did not observe any significant gain after adding Segmentation Mismatch Entity-Type Mismatch Table 7: Representative examples of system errors. Fine-tuned BERTOverflow 61.71 58.75 60.19 SoftNER ( BERTOverflow ) 61.92 60.26 61.08 Table 8: Evaluation on the GitHub NER dataset of readme files and issue posts. All the models are trained on our StackOverflow NER corpus. Our SoftNER model performs close to BiLSTM-CRF model trained on the GitHub ELMo embeddings. the code recognizer and segmenter vectors to the Github ELMo embeddings. We think one likely explanation is that GitHub data contains less code-related tokens when compared to StackOverflow. The percentage of code-related entity tokens is 63.20% in GitHub and 77.21% in StackOverflow. Overall, we observe a drop of our SoftNER tagger from 79.10 F 1 on StackOverflow (Table 2) to 61.08 F 1 on GitHub data (Table 8) in F 1 due to domain mismatch. However, we believe that our NER tagger still achieves sufficient performance to be useful for applications on GitHub. 7 We leave investigation of semi-supervised learning and other domain adaptation approaches for future work. The CoNLL 2003 dataset (Sang and De Meul-der, 2003) is a widely used benchmark for named entity recognition, which contains annotated newswire text from the Reuters RCV1 corpus. State-of-the-art approaches on this dataset (Baevski et al., 2019) use a bidirectional LSTM (Lample et al., 2016; Ma and Hovy, 2016) with conditional random field (Collobert et al., 2011) and contextualized word representations (McCann et al., 2017; Peters et al., 2018; Devlin et al., 2019). Named entity recognition has been explored for new domains and languages, such as social media (Finin et al., 2010; Ritter et al., 2011; Plank et al., 2014; Derczynski et al., 2015; Limsopatham and Collier, 2016; Aguilar et al., 2017), biomedical texts (Collier and Kim, 2004; Greenberg et al., 2018; Kulkarni et al., 2018), multilingual texts (Benajiba et al., 2008; Xie et al., 2018) and code-switched corpora (Aguilar et al., 2018; Ball and Garrette, 2018). Various methods have been investigated for handling rare entities, for example incorporating external context (Long et al., 2017) or approaches that make use of distant supervision (Choi et al., 2018; Yang et al., 2018; Onoe and Durrett, 2019). There has been relatively little prior work on named entity recognition in the software engineering domain. Ye et al. (2016) annotated 4,646 sentences from StackOverflow with five named entity types (Programming Language, Platform, API, Tool-Library-Framework and Software Stan-dard). The authors used a traditional feature-based CRF to recognize these entities. In contrast, we present a much larger annotated corpus consisting of 15,372 sentences labeled with 20 fine-grained entity types. We also develop a novel attention based neural NER model to extract those fine-grained entities. In this work, we investigated the task of named entity recognition in the social computer programming domain. We developed a new NER corpus of 15,372 sentences from StackOverflow and 6,510 sentences from GitHub annotated with 20 fine-grained named entities. We demonstrate that this new corpus is an ideal benchmark dataset for contextual word representations, as there are many challenging ambiguities that often require long-distance context to resolve. We also proposed a novel attention based model, named SoftNER, that outperforms the state-of-the-art NER models on this dataset. Furthermore, we investigated the important sub-task of code recognition. Our code recognition model captures additional spelling information beyond then contextual word representations and consistently helps to improve the NER performance. We believe our corpus, StackOverflow-specific BERT embeddings and named entity tagger will be useful for various language-and-code tasks, such as code retrieval, software knowledge base extraction and automated question-answering. We thank anonymous reviewers for their thoughtful comments. We also thank NVIDIA, Google, and Ohio Supercomputer Center (Center, 2012) for providing GPU/TPU computing resources; Wuwei Lan for kindly helping to train in-domain BERT on StackOverflow data; Sydney Lee, Rita Tong, Lillian Chow, and Raleigh Potluri for help with data annotation. This research is supported in part by the NSF awards IIS-1822754 and IIS-1845670, ODNI and IARPA via the BETTER program contract 19051600004, ARO and DARPA via the SocialSim program contract W911NF-17-C-0095, Criteo Faculty Research Award to Wei Xu, and Amazon Faculty Research Award to Alan Ritter. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the offi-cial policies, either expressed or implied, of NSF, ODNI, IARPA, ARO, DARPA or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
abstain abstain abstain objective method result result abstain abstain abstain other abstain abstain objective abstain abstain objective objective objective abstain objective abstain objective objective result method method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain method abstain abstain method abstain other method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain method method method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain result abstain abstain abstain result other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method method abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain objective objective objective objective objective result method other other other other other