It is very critical to analyze messages shared over social networks for cyber threat intelligence and cyber-crime prevention. In this study, we propose a method that leverages both domain-specific word embeddings and task-specific features to detect cyber security events from tweets. Our model employs a convolutional neural network (CNN) and a long short-term memory (LSTM) recurrent neural network which takes word level meta-embeddings as inputs and incorporates contextual embeddings to classify noisy short text. We collected a new dataset of cyber security related tweets from Twitter and manually annotated a subset of 2K of them. We experimented with this dataset and concluded that the proposed model outperforms both traditional and neural baselines. The results suggest that our method works well for detecting cyber security events from noisy short text. Twitter has become a medium where people can share and receive timely messages on about anything. People share facts, opinions, broadcast news and communicate with each other through these messages. Due to the low barrier to tweeting, and growth in mobile device usage, tweets might provide valuable information as people often share instantaneous updates such as the breaking news before even being broadcasted in the newswire c.f . Petrovic et al. (2010). People also share cyber security events in their tweets such as zero day exploits, ransomwares, data leaks, security breaches, vulnerabilities etc . Automatically detecting such events might have various practical applications such as taking the necessary precautions promptly as well as creating self-awareness as illustrated in Fig. 1. Recently, working with the cyber security  Corresponding author. Dear @AppleSupport, we noticed a *HUGE* security issue at MacOS High Sierra. Anyone can login as root with empty password after clicking on login button several times. Are you aware of it @Apple? related text has garnered a lot of interest in both computer security and natural language processing (NLP) communities ( c.f . Joshi et al. (2013); Ritter et al. (2015); Roy et al. (2017)). Nevertheless, detecting cyber security events from tweets pose a great challenge, as tweets are noisy and often lack sufficient context to discriminate cyber security events due to length limits. Recently, deep learning methods have shown to be outperforming traditional approaches in several NLP tasks (Chen and Manning, 2014; Bahdanau et al., 2014; Kim, 2014; Hermann et al., 2015). Inspired by this progress, our goal is to detect cyber security events in tweets by learning domain-specific word embeddings and task-specific features using neural architectures. The key contribution of this work is two folds. First, we propose an end-to-end learning system to effectively detect cyber security events from tweets. Second, we propose a noisy short text dataset with annotated cyber security events for unsupervised and supervised learning tasks. To our best knowledge, this will be the first study that incorporates domain-specific meta-embeddings and contextual embeddings for detecting cyber security events. In the subsequent sections, we address the challenges to solve our task. The proposed system overview is illustrated in Fig. 2. Preprocessing Normalization Tokenization word2vec GloVe fastText huge flaw   meltdown meltdown ... Word embedding methods might capture different semantic and syntactic features about the same word. To exploit this variety without losing the semantics, we learn meta-embeddings for words. Word Embeddings. Word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), and fastText (Joulin et al., 2016; Bojanowski et al., 2016) are trained for learning domain specific word embeddings on the unlabeled tweet corpus. Meta-Encoder. Inspired by Yin and Schutze (2015) we learn meta-embeddings for words with the aforementioned word embeddings. We use a Convolutional Autoencoder (Masci et al., 2011) for encoding 3 xD size embeddings to a 1 xD dimensional latent variable and to reconstruct the original embeddings from this latent variable. Both encoder and decoder are comprised of 2 convolutional layers where 32 neurons are used on each. The encoder part is shown in Fig. 3. We argue that this network learns a much simpler mapping while capturing the semantic and syntactic relations from each of these embeddings, thus leading to a richer word-level representation. Another advantage of learning meta-embeddings for words Meta-Embedding Vector Convolutional Features 3xD Word Embeddings Figure 3: Convolutional encoder as a feature extractor. is that the proposed architecture alleviates the Out-of-Vocabulary (OOV) embeddings problem, as we still get embeddings from the fastText channel, in contrast to GloVe and word2vec, where no embeddings are available for OOV words. task-specific features from tweets. LDA. Latent Dirichlet Allocation (LDA) is a generative probabilistic model to discover topics from a collection of documents (Blei et al., 2003). LDA works in an unsupervised manner and learns a fi-nite set of categories from a collection, thus represents documents as mixtures of topics. We train an LDA model to summarize each tweet by using the topic with the maximum likelihood e.g . with the topic vulnerability for the tweet in Fig 1. NER. Named Entity Recognition (NER) tags the specified named entities from raw text into pre-defined categories. Named entities could be more general categories such as people, organizations, or specific entities can be learned by creating a dataset containing specific entity tags. We employ an automatically annotated dataset that contains entities from cyber security domain (Bridges et al., 2013) to train our Conditional Random Field model using handcrafted features, i.e. , uni-gram, bi-gram, and gazetteers. The dataset comprises of 850K tokens that contain named entities such as Relevant Term', Operating System',Hardware', Software', Vendor', in the standard IOB-tagging format. Our NER model tags password as Relevant Term' and Apple as Vendor' for the tweet in Fig 1. IE. Uncovering entities and the relations between those entities is an important task for detecting cyber security events. In order to address this we use Information Extraction (IE), in particular OpenIE annotator(Angeli et al., 2015) from the Stanford CoreNLP (Manning et al., 2014). Subsequently, we extract relations between noun phrases with the following dependency triplet (cid:104) arg 1 , rel, arg 2 (cid:105) , where arg 1 , arg 2 denote the arguments and rel represents an implicit semantic relation between those arguments. Hence, the following triplet is extracted from the tweet in Fig. 1, (cid:104) we, noticed, huge security issue (cid:105) . Contextual-Encoder. We use the outputs of LDA, NER and IE algorithms to obtain a combined vector representation using meta-embeddings described in Sec. 2.1. Thus, contextual embeddings are calculated as follows 1 . where  function extracts contextual embeddings and  denotes a tweet, f ,  ,  and  represent meta-embedding, LDA, NER, and IE functions, respectively. Lastly, N and M denote the output tokens. Inspired by the visual question answering task (Antol et al., 2015), where different modalities are combined by CNNs and RNNs, we adopt a similar network architecture for our task. Prior to training, and inference we preprocess, normalize and tokenize each tweet as described in Sec. 3. CNN. We employ a CNN model similar to that of (Kim, 2014) where we feed the network with static meta-embeddings. Our network is comprised of one convolutional layer with varying filter sizes, that is 2 , 3 , 5 . All tweets are zero padded to the maximum tweet length. We use ReLU as activation and global max pooling at the end of CNN. RNN. We use a bi-directional LSTM (Hochreiter and Schmidhuber, 1997) and read the input in both directions and concatenate forward and backward hidden states to encode the input as a sequence. Our LSTM model is comprised of a single layer and employs 100 neurons. Data Collection. We collected 2 . 5 M tweets using the Twitter's streaming API over a period from 2015-01-01 to 2017-12-31 using an initial 1 We used zero vectors for the non-existent relations. set of keywords, henceforth referred as seed keywords to retrieve cyber security related tweets. In particular, we use the main group names of cyber security taxonomy described in Le Sceller et al. (2017) as seed keywords e.g. denial of ser-vice', botnet', malware', vulnerability', phish-ing', data breach' to retrieve relevant tweets. Using seed keywords is a practical way to filter out noise considering sparsity of cyber security related tweets in the whole tweet stream. After the initial retrieval, we use langid.py (Lui and Baldwin, 2012) to filter out non-English tweets. Data Preprocessing. We substitute user handles with $ mention $ , and hyperlinks with $ url $ . We remove emoticons and reserved keyword RT which denotes retweets. We substitute hashtags by removing the prefix # character. We limit characters that repeat more than two times, remove capitalization and tokenize tweets using the Twitter tokenizer in nltk library. We normalize nonstandard forms, i.e . writing cu tmrrw instead of see you tomorrow . Although there are several reasons for that, the most prominent one is that people tend to mimic prosodic effects in speech (Eisen-stein, 2013). To overcome this, we use lexical normalization, where we substitute OOV tokens with in-Vocabulary (IV) standard forms, i.e . a standard form available in a dictionary. In particular we use UniMelb (Han et al., 2012), UTDallas (Liu et al., 2011) datasets. Lastly, we remove identical tweets and check the validity by removing tweets with less than 3 non-special tokens. Data Annotation. We instructed cyber security domain experts for manual labelling of the dataset. Annotators are asked to provide a binary label for whether there is a cyber security event in the given tweet or not. Annotators are told to skip tweets if they are unsure about their decisions. Finally, we validated annotations by only accepting annotations if at least 3 among 4 annotators agreed on. Therefore, we presume the quality of attained ground truth labels is dependable. Overall, 2 K tweets are annotated. Dataset Statistics. After preprocessing, our initial 2 . 5 M tweet dataset is reduced to 1 . 7 M tweets where 2 K of them are labeled 2 . The labeled dataset is somewhat balanced as there are 843 event-related tweets and 1157 non-event tweets. The training and testing sets have 1600 and 400 samples, respectively. 2 Available at http://stmai.github.io/cydec Training. We used Keras with Tensorflow back-end in our neural models. For fastText and word2vec embeddings we used Gensim, and for GloVe we used glove-python library. For training the word embeddings, we use the entire tweet text corpus and obtain 100 dimensional word embeddings. We set word2vec and fastText model's alpha parameter to 0 . 025 and window size to 5 . For GloVe embedding model, we set the learning rate to 0 . 01 , alpha to 0 . 75 and maximum count parameter to 100 . For embedding models, we determined the minimum count parameter to 5 , culminating in the elimination of infrequent words. Consequently, we have 3 , 100 -dimensional word embedding tensor in which first, second and third channels consist of word2vec, fastText and GloVe embeddings respectively. We then, encode these 3 x 100 dimensional embeddings into 1 x 128 dimensional representations by using our Meta-Encoder. We train our two channel architecture that combines both LSTM and CNN with 2 inputs: meta-embeddings and contextual embeddings. We use meta-embeddings for feature learning via LSTM and CNN, and their feature maps are concatenated with contextual embeddings in the Fusion Layer. In the end, fully connected layers and a softmax classifier are added, and the whole network is trained to minimize binary cross entropy loss with a learning rate of 0.01 by using the Adam optimizer (Kingma and Ba, 2014). 3 Baselines. To compare with our results, we implemented the following baselines: SVM with BoW : We trained an SVM classifier using Bag-of-words (BoW) which provides a simplified representation of textual data by calculating the occurrence of words in a document. SVM with meta-embeddings : We trained an SVM classifier with the aforementioned meta-embeddings. CNN-Static : We used Kim (2014)'s approach using word2vec embeddings. Results. Table 1 summarizes the overall performance of each method. To compare the models, we used four different metrics: accuracy, recall, precision and F1-score. Each reported result is the mean of a 5-fold cross validation experiment. It is clear that our method outperforms various simple and neural baselines. Also, in Table 2, we provide results of our proposed model along with the ground-truth annotations. We also provide results with the different combinations of contextual fea-3 See supplementary for hyperparameter choices. Human Study. 8 different subjects are thoroughly instructed about what is considered as a cyber security event and individually asked to label 50 randomly selected tweets from the test set. The results are provided in Table 3. Error Analysis. In order to understand how our system performs, we randomly select a set of erroneously classified instances from the test dataset. Type I Errors. Our model identifies this tweet as an event uk warned following breach in air pollution regulation $url$ whereas it is clearly about the a breach of a regulation. We hypothesize that this is due to the lack of sufficient training data. Following tweet is also identified as an event wannacry ransomware ransomwareattack ransomwarewannacry malware $url$. We suspect that the weights of multiple relevant terms deceive the model. Type II Errors. Our model fails to identify the following positive sample as an event. For playsta-tion network was the target of miraibotnet ddos attack guiding tech rss news feed search our model fails to recognize the 'miraibotnet' from the tweet. We suspect this is due to the lack of hashtag decomposition; otherwise, the model could recognize mirai' and botnet' as separate words. Discussions. Cyber security related tweets are complicated and analysing them requires in-depth domain knowledge. Although human subjects are properly instructed, the results of the human study indicate that our task is challenging and humans can hardly discriminate cyber security events amongst cyber security related tweets. To further investigate this, we plan to increase the number of human subjects. One limitation of this study is that we do not consider hyperlinks and user handles which might provide additional information. One particular problem we have not addressed in this work is hashtag decomposition. Error analysis indicates that our model might get confused by challenging examples due to ambiguities and lack of context. 4 See supplementary for feature combination details. Event detection on Twitter is studied extensively in the literature (Petrovic et al., 2010; Sakaki et al., 2010; Weng and Lee, 2011; Ritter et al., 2012; Yuan et al., 2013; Atefeh and Khreich, 2015). Banko et al. (2007) proposed a method to extract relational tuples from web corpus without requiring hand labeled data. Ritter et al. (2012) proposed a method for categorizing events in Twitter. Luo et al. (2015) suggested an approach to infer binary relations produced by open IE systems. Recently, Ritter et al. (2015) introduced the first study to extract event mentions from a raw Twitter stream for event categories DDoS attacks, data breaches, and account hijacking. Chang et al. (2016) proposed an LSTM based approach which learns tweet level features automatically to extract events from tweet mentions. Lately, Le Sceller et al. (2017) proposed a model to detect cyber security events in Twitter which uses a taxonomy and a set of seed keywords to retrieve relevant tweets. Tonon et al. (2017) proposed a method to detect events from Twitter by using semantic analysis. Roy et al. (2017) proposed a method to learn domain-specific word embeddings for sparse cyber security text. Prior art in this direction (Ritter et al., 2015; Chang et al., 2016) focuses on extracting events and in particular predicting the events' posterior given the presence of particular words. Le Sceller et al. (2017); Tonon et al. (2017) focus on detecting cyber security events from Twitter. Our work distinguishes from prior studies as we formulate cyber security event detection problem as a classification task and learn meta-embeddings from domain-specific word embeddings while incorporating task-specific features and employing neural architectures. We introduced a novel neural model that utilizes meta-embeddings learned from domain-specific word embeddings and task-specific features to capture contextual information. We present a unique dataset of cyber security related noisy short text collected from Twitter. The experimental results indicate that the proposed model outperforms the traditional and neural baselines. Possible future research direction might be detecting cyber security related events in different languages. We would like to thank Merve Nur Ylmaz and Benan Bardak for their invaluable help with the annotation process on this project. This research is fully supported by STM A.S. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of the sponsor.
abstain objective method objective objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective objective objective method abstain abstain abstain method abstain abstain abstain method method abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain method method method abstain method abstain abstain abstain method method abstain method method abstain method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain other other other other other other other other other other other method objective method abstain abstain other other other