Breaking cybersecurity events are shared across a range of websites, including security blogs (FireEye, Kaspersky, etc.), in addition to social media platforms such as Face-book and Twitter. In this paper, we investigate methods to analyze the severity of cybersecurity threats based on the language that is used to describe them online. A corpus of 6,000 tweets describing software vulnerabilities is annotated with authors' opinions toward their severity. We show that our corpus supports the development of automatic classifiers with high precision for this task. Furthermore, we demonstrate the value of analyzing users' opinions about the severity of threats reported online as an early indicator of important software vulnerabilities. We present a simple, yet effective method for linking software vulnerabilities reported in tweets to Common Vulnerabilities and Exposures (CVEs) in the National Vulnerability Database (NVD). Using our predicted severity scores, we show that it is possible to achieve a Precision@50 of 0.86 when forecasting high severity vulnerabilities, significantly outperforming a baseline that is based on tweet volume. Finally we show how reports of severe vulnerabilities online are predictive of real-world exploits. 1 1 Introduction Software vulnerabilities are flaws in computer systems that leave users open to attack; vulnerabilities are generally unknown at the time a piece of software is first published, but are gradually identified over time. As new vulnerabilities are discovered and verified they are assigned CVE numbers (unique identifiers), and entered into the National Vulnerability Database (NVD). 2 To help prioritize 1 Our code and data are available at https://github.com/viczong/ cybersecurity threat severity analysis . response efforts, vulnerabilities in the NVD are assigned severity scores using the Common Vulnerability and Scoring System (CVSS). As the rate of discovered vulnerabilities has increased in recent years, 3 the need for efficient identification and pri-oritization has become more crucial. However, it is well known that a large time delay exists between the time a vulnerability is first publicly disclosed to when it is published in the NVD; a recent study found that the median delay between the time a vulnerability is first reported online and the time it is published in the NVD is seven days; also, 75% of threats are first disclosed online giving attackers time to exploit the vulnerability. 4 In this paper we present the first study of whether natural language processing techniques can be used to analyze users' opinions about the severity of software vulnerabilities reported online. We present a corpus of 6,000 tweets annotated with opinions toward threat severity, and empirically demonstrate that this dataset supports automatic classification. Furthermore, we propose a simple, yet effective method for linking software vulnerabilities reported on Twitter to entries in the NVD, using CVEs found in linked URLs. We then use our threat severity analyzer to conduct a large-scale study to validate the accuracy of users' opinions online against experts' severity 3 https://www.cvedetails.com/browse-by-date.php 4 https://www.recordedfuture.com/vulnerability-disclosure-delay/ ratings (CVSS scores) found in the NVD. Finally, we show that our approach can provide an early indication of vulnerabilities that result in real exploits in the wild as measured by the existence of Symantec virus signatures associated with CVEs; we also show how our approach can be used to retrospectively identify Twitter accounts that provide reliable warnings about severe vulnerabilities. Recently there has been increasing interest in developing NLP tools to identify cybersecurity events reported online, including denial of service attacks, data breaches and more (Ritter et al., 2015; Chang et al., 2016; Chambers et al., 2018). Our proposed approach in this paper builds on this line of work by evaluating users opinions toward the severity of cybersecurity threats. Prior work has also explored forecasting software vulnerabilities that will be exploited in the wild (Sabottke et al., 2015). Features included structured data sources (e.g., NVD), in addition to the volume of tweets mentioning a list of 31 keywords. Rather than relying on a fixed set of keywords, we analyze message content to determine whether the author believes a vulnerability is severe. As discussed by Sabottke et al. (2015), methods that rely on tracking keywords and message volume are vulnerable to adversarial attacks from Twitter bots or sockpuppet accounts (Solorio et al., 2013). In contrast, our method is somewhat less prone to such attacks; by extracting users' opinions expressed in individual tweets, we can track the provenance of information associated with our forecasts for display to an analyst, who can then determine whether or not they trust the source of information. Given a tweet t and named entity e , our goal is to predict whether or not there is a serious cybersecurity threat towards the entity based on context. For example, given the context in Figure 2, we aim at predicting the severity level towards adobe flash player . We define an author's perceived severity toward a threat using three criteria: (1) does the author believe that their followers should be worried about the threat? (2) is the vulnerability easily exploitable? and (3) could the threat affect a large number of users? If one or more of these criteria are met, then we consider the threat to be severe. To collect tweets describing cybersecurity events for annotation, we tracked the keywords ddos and vulnerability from Dec 2017 to July 2018 using the Twitter API. We then used the Twitter tagging tool described by Ritter et. al. (2011) to extract named entities, 5 retaining tweets that contain at least one named entity. To cover as many linguistic variations as possible, we used Jaccard similarity with a threshold of 0.7 to identify and remove duplicated tweets with same date. 6 2.2 Mechanical Turk Annotation We paid crowd workers on Amazon Mechanical Turk to annotate our dataset. The annotation was performed in two phases; during the first phase, we asked workers to determine whether or not the tweet describes a cybersecurity threat toward a target entity, in the second phase the task is to determine whether the author of the tweet believes the threat is severe; only tweets that were judged to express a threat were annotated in the second phase. Each HIT contained 10 tweets to be annotated; workers were paid $0.20 per HIT. In pilot studies we tried combining these two annotations into a single task, but found low inter-rater agreement, especially for the threat severity judgments, motivating the need for separation of the annotation procedure into two tasks. Figure 2 shows a portion of the annotation interface presented to workers during the second phase of annotation. Details of each phase are described below, and summarized in Table 1. Threat existence annotation: Not all tweets in our dataset describe cybersecurity threats, for example many tweets discuss different senses of the 5 https://github.com/aritter/twitter nlp 6 We sampled a dataset of 6,000 tweets to annotate. word vulnerability (e.g., It's OK to show vul-nerability). During the first phase of our annotation process, workers judged whether or not there appears to be a cybersecurity threat towards the target entity based on the content of the corresponding tweet. We provide workers with 3 options: the tweet indicates (a) a cybersecurity threat towards given entity, (b) a threat, but not towards the target entity, or (c) no cybersecurity threat. Each tweet is annotated by 5 workers. Threat severity annotation: In the second phase, we collect all tweets judged to contain threats by more than 3 workers in the first phase and annotated them for severity. 1,966 tweets were selected out of 6,000. 7 For each tweet we provided workers with 3 options: the tweet contains (a) a severe, (b) a moderate or (c) no threat toward the target entity. During our pilot study, we found this to be a more challenging annotation task, therefore we increased the number of annotators per tweet to 10 workers, which we found to improve agreement with our expert judgments. Inter-annotator agreement: During both phases, we monitored the quality of workers' annotations using their agreement with each other. We calculated the annotation agreement of each worker against the majority vote of other workers. We manually removed data from workers who have an agreement less than 0.5, filling in missing annotations with new workers. We also manually removed data from workers who answered either uniformly or randomly for all HITs. Agreement with expert judgments: To validate the quality of our annotated corpus we compared the workers' aggregated annotations against our own expert annotations. We independently annotated 150 randomly sampled tweets, 61 tweets of 7 We further deduplicate pairs of tweets where the longest common subsequence covers the majority of the text contents. During deduplication all hashtags and URLs were removed and digits were replaced with 0. which are marked as containing severe or moderate threats. For threat existence annotation, we observe a 0.66 value of Cohen's  (Artstein and Poesio, 2008) between the expert judgements and majority vote of 5 crowd workers. Although our threat severity annotation task may require some cybersecurity knowledge for accurate judgment, we still achieve 0.52 Cohen  agreement by comparing majority vote from 10 workers with expert annotations. Using the annotated corpus described in Section 2.2, we now develop classifiers that detect threats reported online and analyze users' opinions toward their severity. Specifically, given a named entity and tweet, (cid:104) e, t (cid:105) , our goal is to estimate the probability the tweet describes a cybersecurity threat towards the entity, p threat ( y |(cid:104) e, t (cid:105) ) and also the probability that the threat is severe, p severe ( y |(cid:104) e, t (cid:105) ) . In this section, we describe the details of these classifiers and evaluate their performance. We experimented with two baselines to detect reports of cyberthreats and analyze opinions about their severity: logistic regression using bag-of-ngram features, and 1D convolutional neural networks. In the sections below we describe the input representations and details of these two models. Logistic regression: We use logistic regression as our first baseline model for both classifiers. Input representations are bag-of-ngram features extracted from the entire tweet content. Example features are presented in Table 4. We use context windows of size 2, 3 and 4 to extract features. We map extracted n -grams that occur only once to a (cid:104) UNK (cid:105) token. In all our experiments, we replace named entities with a special token (cid:104) TARGET (cid:105) ; this helps prevent our models from biasing towards specific entities that appear in our training corpus. All digits are replaced with 0. Convolutional neural networks: We also experimented with 1D convolutional neural networks (Collobert et al., 2011; Kim, 2014). Given a tweet, the model first applies convolutional operations on input sequences with various filters of different sizes. The intermediate representations for each filter are aggregated using max-pooling over time, followed by a fully connected layer. We choose convolution kernel sizes to be 3, 4 and 5-grams with 100 filters for each. We minimize cross-entropy loss using Adam (Kingma and Ba, 2015); the learning rate is set to 0.001 with a batch size of 1 and 5 epochs. Word embeddings: We train our own cybersecurity domain word embeddings based on GloVe (Pennington et al., 2014), as 39.7% of our tokens are treated as OOV words in GloVe pre-trained Twitter embeddings. We used a corpus of 609,470 cybersecurity-related tweets (described in Section 2.1) as our training corpus. The dimension of word embeddings is 50. Table 2 shows nearest neighbors for some sampled cybersecurity terms based on the learned embeddings. During network training, we initialize word embedding layer with our own embeddings. We initialize tokens not in our trained embeddings by randomized vectors with uniform distribution from -0.01 to 0.01. We fine-tune the word embedding layer during training. For threat existence classification, we randomly split our dataset of 6,000 tweets into a training set of 4,000 tweets, a development set of 1,000 tweets, and test set of 1,000 tweets. For the threat severity classifier, we only used data from 2nd phase of annotation. This dataset consists of 1,966 tweets that were judged by the mechanical turk workers to describe a cybersecurity threat towards the target entity. We randomly split this dataset into a training set of 1,200 tweets, a development set of 300 tweets, and a test set of 466 tweets. We collapsed the three annotated labels into two categories based on whether or not the author expresses an opinion that the threat towards the target entity is severe. Threat existence classifier: The logistic regression baseline has good performance at identifying threats, which we found to be a relatively easy task; area under the precision-recall curve (AUC) on the development and test set presented in Table 5. This enables accurate detection of trending threats online by tracking cybersecurity keywords using the Twitter streaming API, following an approach that is similar to prior work on entity-based Twitter event detection (Ritter et al., 2012; Zhou et al., 2014; Wei et al., 2015). Table 3 presents an example of threats detected using this procedure on Nov. 22, 2018. 8 Threat severity classifier: Figure 3 shows precision recall curves for the threat severity classifiers. Logistic regression with bag-of-ngram features provides a strong baseline for this task. Table 4 presents examples of high-weight features from the logistic regression model. These features often intuitively indicate severe threats, e.g. critical vulnerability, a massive, million, etc. Without much hyperparameter tuning on the development set, the convolutional neural network consistently achieves higher precision at the same level of recall as compared to logistic regression. We summarize the performance of our threat existence and severity classifiers in Table 5. 3 Forecasting Severe Cybersecurity Threats In Section 2 we presented methods that can accurately detect threats reported online and analyze users' opinions about their severity. We now explore the effectiveness of this model for forecasting. Specifically, we aim to answer the following questions: (1) To what extent do users' opinions about threat severity expressed online align with expert judgments? (2) Can these opinions provide an early indicator to help prioritize threats based 8 A live demo is available at: http://kb1.cse.ohio-state.edu:8123/ events/threat Named Entity Example Tweet Existence Severity apple RT AsturSec: A kernel vulnerability in Apple devices gives access to remote code execution Packt Hub #infosec #CyberSecurity https://t.... A large corpus of users' opinions: We follow the same procedure described in Section 2.1 to prepare another dataset for a large-scale evaluation. For this purpose, we collected data from Jan 2016 to Nov 2017; this ensures no tweets overlap with those that were annotated in Section 2.2. We collect all English tweets that explicitly contain the keyword vulnerability within this time period, which results in a total number of 976,180 tweets. 377,468 tweets remain after removing tweets without named entities. National Vulnerability Database (NVD): NVD is the U.S. government database of software vulnerabilities. Started in 2000, NVD covers over 100,000 vulnerabilities, assigning a unique CVE number for each threat. These CVE numbers serve as common identifiers. NVD uses the Common Vulnerability Scoring System (CVSS) to measure the severity of threats. CVSS currently has two versions: CVSS v2.0 and CVSS v3.0 standards. CVSS v3.0 is the latest version released in July 2015. We summarize the two standards in Table 6. 9 Severity Base Score Severity Base Score None 0.0 Low 0.0-3.9 Low 0.1-3.9 Medium 4.0-6.9 Medium 4.0-6.9 High 7.0-10.0 High 7.0-8.9 Critical 9.0-10.0 Table 6: Qualitative severity rankings of vulnerabilities in NVD. relies on accurately matching tweets describing vulnerabilities to their associated NVD records. To achieve this we present a simple, yet effective method that makes use of content in linked web-pages. We find that 82.4% of tweets contain external urls in our dataset. Our approach to link tweets to CVEs is to search for CVE numbers either in url addresses or in corresponding web pages linked in tweets reporting vulnerabilities. 10 We ignore web pages that contain more than one unique CVE to avoid potential ambiguities. Using this approach, within our dataset, 79,383 tweets were linked to 10,565 unique CVEs. In order to stimulate a forecasting scenario, we only consider CVEs where more than two associated tweets were posted at least 5 days ahead of official NVD publication date. In our dataset, 13,942 tweets are finally selected for forecast evaluation, covering 1,409 unique CVE numbers. To evaluate the accuracy of this linking procedure, we randomly sampled 100 matched pairs and manually checked them. We find the precision of our matching procedure to be very high: only 2 mismatches out of 100 are found. Now that we have a linking between tweets and CVE numbers, our goal is to produce a sorted list of CVEs with those that are indicated to be severe threats the top. We consider two ranking procedures, detailed below; the first is based on users' opinions toward the severity of a threat, and the second is a baseline that simply uses the volume of tweets describing a specific vulnerability to measure its severity. To simplify the exposition below, we denote each CVE number as CVE i , and the collection of tweets linked to this CVE number as TCVE i = { k | tweet t k is mapped to CVE i } . Our model: Our severe threat classifier assigns a severity score p severity ( y |(cid:104) e, t (cid:105) ) for each tuple of name entity e and corresponding tweet t . For a specific CVE, we define our severity forecast score to be the maximum severity scores among all tuples from matched tweets (cid:104) , t k (cid:105) (a single tweet 10 Readers may be wondering why a CVE number has been generated before it is officially published in the database. This is due to the mechanism of assigning CVEs. Some identified companies have the right to assign CVEs or have already reversed some CVEs. When a threat appears, a CVE number is assigned immediately before any further evaluation. NVD only officially publishes a threat after all evaluations are completed. Therefore, there is a time delay between CVE entry established date and the official publication date. may contain more than one name entity): ( CVE i ) forecast score = max k  TCVE i p severity ( y |(cid:104) , t k (cid:105) ) . Tweet volume baseline: Intuitively, the number of tweets and retweets can indicate people's concern about a specific event. Specifically, the severity for threat CVE i according to the volume model is defined by the cardinality of TCVE i : ( CVE i ) volume score = | TCVE i | . In our first set of experiments, we compare our forecasted threat severity scores against CVSS ratings from the NVD. We define a threat as being severe if its CVSS score is  7.0. This cut-off corresponds to qualitative severity ratings provided by CVSS (marked as HIGH or CRITICAL in Table 6). 11 We use the newest v3.0 scoring system, which was developed to improve v2.0. 12 Large software vendors have announced of the adaptation of the CVSS v3.0 standards, including Cisco, Oracle, SUSE Linux, and RedHat. We evaluate our models' performance at identifying severe threats five days ahead of the NVD publication date, within their top k predictions. Table 7 shows our results. We observe that tweet volume performs better than a random baseline; having a large number of tweets beforehand is a good indicator for high severity, however our approach which analyzes the content of messages discussing software vulnerabilities achieves significantly better performance; 86% of its top 50 forecasts were indeed rated as HIGH or CRITICAL severity in the NVD. 11 The Forum of Incident Response and Security Teams (FIRST) also provides an example guideline that recommends patching all vulnerabilities with CVSS scores  7.0. See https://www.first.org/cvss/cvss-based-patch-policy.pdf . 12 https://www.first.org/cvss/user-guide CVE Num / Name Entity CVE Description / Matched Tweets CVSS Scores / Our Severity Publish Date (# Days Ahead) Table 8 presents top 4 forecast results from our model. We observe that our model can predict accurate severity level even 19 days ahead of the official published date in NVD (Table 8(a), (c)). In addition to comparing our forecasted severity scores against CVSS, as described above, we also explored several alternatives suggested by the security community to evaluate our methods: (1) Symantec's anti-virus (AV) signatures 13 and intrusion-protection (IPS) signatures, 14 in addition to (2) Exploit Database (EDB). 15 Sabottke et al. (2015) suggested Symantec's AV and IPS signatures are the best available indicator 13 https://www.symantec.com/security-center/a-z 14 https://www.symantec.com/security response/attacksignatures/ 15 https://www.exploit-db.com/ for real exploitable threats in the wild. We follow their method of explicitly querying for CVE numbers from the descriptions of signatures to generate exploited threats ground truth. Exploit Database (EDB) is an archive of public exploits and software vulnerabilities. We query EDB for all threats that have been linked into NVD. 16 In total we gathered 134 CVEs verified by Symantec and EDB to be real exploits within the 1,409 CVEs used in our forecasting evaluation. We evaluate the number of exploited threats identified within our top ranked CVEs. Table 9 presents our results. We observe that 7 of top 10 threats from our model were exploited in the wild. We also observe that for the actual CVSS v3.0 scores, only 1 out of the top 10 vulnerabilities was 16 http://cve.mitre.org/data/refs/refmap/source-EXPLOIT-DB.html exploited. Finally we perform an analysis of the reliability of individual Twitter accounts. We evaluate all accounts with more than 5 tweets exceeding 0.5 confidence score from our severity classifier. Table 10 presents our results. Accounts in our data whose warnings were found to have highest precision when compared against CVSS include @se-curityaffairs and @EduardKovacs, which are known to post security related information, and both have more than 10k followers. There is a long history of prior work on analyzing users' opinions online (Wiebe et al., 2004), a large body of prior work has focused on sentiment analysis (Pang et al., 2002; Rosenthal et al., 2015), e.g., determining whether a message is positive or negative. In this paper we developed annotated corpora and classifiers to analyze users' opinions toward the severity of cybersecurity threats reported online, as far as we are aware this is the first work to explore this direction. Forecasting real-world exploits is a topic of interest in the security community. For example, Bozorgi et al. (2010) train SVM classifiers to rank the exploitability of threats. Several studies have also predicted CVSS scores from various sources including text descriptions in NVD (Han et al., 2017; Bullough et al., 2017). Prior work has also explored a variety of forecasting methods that incorporate textual evidence (Smith, 2010), including the use of Twitter message content to forecast influenza rates (Paul et al., 2014), predicting the propagation of social media posts based on their content (Tan et al., 2014) and forecasting election outcomes (O'Connor et al., 2010; Swamy et al., 2017). In this paper, we presented the first study of the connections between the severity of cybersecurity threats and language that is used to describe them online. We annotate a corpus of 6,000 tweets describing software vulnerabilities with authors' opinions toward their severity, and demonstrated that our corpus supports the development of automatic classifiers with high precision for this task. Furthermore, we demonstrate the value of analyzing users' opinions about the severity of threats reported online as an early indicator of important software vulnerabilities. We presented a simple, yet effective method for linking software vulnerabilities reported in tweets to Common Vulnerabilities and Exposures (CVEs) in the National Vulnerability Database (NVD). Using our predicted severity scores, we show that it is possible to achieve a Precision@50 of 0.86 when forecasting high severity vulnerabilities, significantly outperforming a baseline that is based on tweet volume. Finally we showed how reports of severe vulnerabilities online are predictive of real-world exploits. We thank our anonymous reviewers for their valuable feedback. We also thank Tudor Dumitras for helpful discussion on identifying real exploited threats. Funding was provided by the the Office of the Director of National Intelligence (ODNI) and Intelligence Advanced Research Projects Activity (IARPA) via the Air Force Research Laboratory (AFRL) contract number FA8750-16-C0114, in addition to the Defense Advanced Research Projects Agency (DARPA) via the U.S. Army Research Office (ARO) and under Contract Number W911NF-17-C-0095, in addition to an Amazon Research Award and an NVIDIA GPU grant. The content of the information in this document does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred. The U.S. Government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notation here on.
abstain objective abstain result objective method result result abstain abstain other abstain abstain abstain objective objective objective other result abstain objective abstain abstain method abstain objective objective objective method abstain abstain method method method abstain method abstain abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method method method abstain method abstain abstain method method method method abstain abstain method method method method method abstain method method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain other abstain other abstain abstain abstain abstain abstain other method abstain abstain abstain method abstain abstain other method abstain abstain abstain other objective other other other other objective objective objective method result result other other other other other