HOST-BASED INTRUSION DETECTION SYSTEMS
In computer security, designing a robust intrusion detection system is one of the most fundamental and important problems. In this paper, we propose a system-call language-modeling approach for designing anomaly-based host intrusion detection systems. To remedy the issue of high false-alarm rates commonly arising in conventional methods, we employ a novel ensemble method that blends multiple thresholding classifiers into a single one, making it possible to accumulate ‘highly normal’ sequences. The proposed system-call language model has various advantages leveraged by the fact that it can learn the semantic meaning and interactions of each system call that existing methods cannot effectively consider. Through diverse experiments on public benchmark datasets, we demonstrate the validity and effectiveness of the proposed method. Moreover, we show that our model possesses high portability, which is one of the key aspects of realizing successful intrusion detection systems.
In computer security, designing a robust intrusion detection system is one of the most fundamental and important problems. In this paper, we propose a system-call language-modeling approach for designing anomaly-based host intrusion detection systems. To remedy the issue of high false-alarm rates commonly arising in conventional methods, we employ a novel ensemble method that blends multiple thresholding classifiers into a single one, making it possible to accumulate ‘highly normal’ sequences. The proposed system-call language model has various advantages leveraged by the fact that it can learn the semantic meaning and interactions of each system call that existing methods cannot effectively consider. Through diverse experiments on public benchmark datasets, we demonstrate the validity and effectiveness of the proposed method. Moreover, we show that our model possesses high portability, which is one of the key aspects of realizing successful intrusion detection systems.
1 INTRODUCTION
An intrusion detection system (IDS) refers to a hardware/software platform for monitoring network or system activities to detect malicious signs therefrom. Nowadays, practically all existing computer systems operate in a networked environment, which continuously makes them vulnerable to a variety of malicious activities. Over the years, the number of intrusion events is significantly increasing across the world, and intrusion detection systems have already become one of the most critical components in computer security. With the explosive growth of logging data, the role of machine learning in effective discrimination between malicious and benign system activities has never been more important.
A survey of existing IDS approaches needs a multidimensional consideration. Depending on the scope of intrusion monitoring, there exist two main types of intrusion detection systems: networkbased (NIDS) and host-based (HIDS). The network-based intrusion detection systems monitor communications between hosts, while the host-based intrusion detection systems monitor the activity on a single system. From a methodological point of view, intrusion detection systems can also be classified into two classes (Jyothsna et al., 2011): signature-based and anomaly-based. The signaturebased approaches match the observed behaviors against templates of known attack patterns, while the anomaly-based techniques compare the observed behaviors against an extensive baseline of normal behaviors constructed from prior knowledge, declaring each of anomalous activities to be an attack. The signature-based methods detect already known and learned attack patterns well but have an innate difficulty in detecting unfamiliar attack patterns. On the other hand, the anomaly-based methods can potentially detect previously unseen attacks but may suffer from making a robust baseline of normal behavior, often yielding high false alarm rates. The ability to detect a ‘zero-day’ attack (i.e., vulnerability unknown to system developers) in a robust manner is becoming an important requirement of an anomaly-based approach. In terms of this two-dimensional taxonomy, we can classify our proposed method as an anomaly-based host intrusion detection system.
∗To whom correspondence should be addressed.
It was Forrest et al. (1996) who first started to use system-call traces as the raw data for hostbased anomaly intrusion detection systems, and system-call traces have been widely used for IDS research and development since their seminal work (Forrest et al., 2008). Recently, Creech & Hu (2014) proposed to use neural networks on top of a sequence of system calls in the context of HIDS. System calls represent low-level interactions between programs and the kernel in the system, and many researchers consider system-call traces as the most accurate source useful for detecting intrusion in an anomaly-based HIDS. From a data acquisition point of view, system-call traces are easy to collect in a large quantity in real-time. Our approach described in this paper also utilizes system-call traces as input data.
For nearly two decades, various research has been conducted based on analyzing system-call traces. Most of the existing anomaly-based host intrusion detection methods typically aim to identify meaningful features using the frequency of individual calls and/or windowed patterns of calls from sequences of system calls. However, such methods have limited ability to capture call-level features and phrase-level features simultaneously. As will be detailed shortly, our approach tries to address this limitation by generating a language model of system calls that can jointly learn the semantics of individual system calls and their interactions (that can collectively represent a new meaning) appearing in call sequences.
In natural language processing (NLP), a language model represents a probability distribution over sequences of words, and language modeling has been a very important component of many NLP applications, including machine translation (Cho et al., 2014; Bahdanau et al., 2014), speech recognition (Graves et al., 2013), question answering (Hermann et al., 2015), and summarization (Rush et al., 2015). Recently, deep recurrent neural network (RNN)-based language models are showing remarkable performance in various tasks (Zaremba et al., 2014; Jozefowicz et al., 2016). It is expected that such neural language models will be applicable to not only NLP applications but also signal processing, bioinformatics, economic forecasting, and other tasks that require effective temporal modeling.
Motivated by this performance advantage and versatility of deep RNN-based language modeling, we propose an application of neural language modeling to host-based introduction detection. We consider system-call sequences as a language used for communication between users (or programs) and the system. In this view, system calls and system-call sequences correspond to words and sentences in natural languages, respectively. Based on this system-call language model, we can perform various tasks that comprise our algorithm to detect anomalous system-call sequences: e.g., estimation of the relative likelihood of different words (i.e., system calls) and phrases (i.e., a window of system calls) in different contexts.
The idea of using artificial neural networks for IDSs has been popular (Debar et al., 1992; Ryan et al., 1998; Mukkamala et al., 2002; Wang et al., 2010; Creech & Hu, 2014). For more recent deep learning-based techniques, there exists an example that utilized LSTM for improving intrusion detection performance (Staudemeyer & Omlin, 2013; Staudemeyer, 2015). However, the work by Staudemeyer & Omlin (2013); Staudemeyer (2015) was in essence a feature-based supervised classifier (rather than an anomaly detector) requiring heavy annotation efforts to create labels. As such, their work required explicitly labeled attack data and possessed an inherent limitation that it could not detect new types of attacks. In addition, their approach was not an end-to-end framework and needed careful feature engineering to extract salient features for the classification task. Only one binary label was given per sequence to train their model, unlike our proposed method that is trained to predict the next call, effectively capturing contextual information needed for classification.
Our specific contributions can be summarized as follows: First, to model sequences of system calls, we propose a neural language modeling technique that utilizes long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) units for enhanced long-range dependence learning. The present work is one of the first end-to-end frameworks to model system-call sequences as a natural language for effectively detecting anomalous patterns therefrom. Second, to reduce false-alarm rates of anomaly-based intrusion detection, we propose a leaky rectified linear units (ReLU) (Maas et al., 2013) based ensemble method that constructs an integrative classifier using multiple (relatively weak) thresholding classifiers. Each of the component classifiers is trained to detect different types of ‘highly normal’ sequences (i.e., system call sequences with very high probability of being normal), and our ensemble method blends them to produce a robust classifier that delivers significantly lower false-alarm rates than other commonly used ensemble methods. As shown in Figure 1, these
two aspects of our contributions can seamlessly be combined into a single framework. Note that the ensemble method we propose is not limited to our language-model based front-end but also applicable to other types of front-ends.
In the rest of this paper, we will explain more details of our approach and then present our experimental results that demonstrate the effectiveness of our proposed method.
2 PROPOSED METHOD
Figure 1 shows the overview of our proposed approach to designing an intrusion detection system. Our method consists of two parts: the front-end is for language modeling of system calls in various settings, and the back-end is for anomaly prediction based on an ensemble of thresholding classifiers derived from the front-end. In this section, we describe details of each component in our pipeline.
2.1 LANGUAGE MODELING OF SYSTEM CALLS
Figure 2 illustrates the architecture of our system-call language model. The system call language model estimates the probability distribution of the next call in a sequence given the sequence of previous calls. We assume that the host system generates a finite number of system calls. We index each system call by using an integer starting from 1 and denote the fixed set of all possible system calls in the system as S = {1, · · · ,K}. Let x = x1x2 · · ·xl(xi ∈ S) denote a sequence of l system calls.
At the input layer, the call at each time step xi is fed into the model in the form of one-hot encoding, in other words, a K dimensional vector with all elements zero except position xi. At the embedding layer, incoming calls are embedded to continuous space by multiplying embedding matrix W , which should be learned. At the hidden layer, the LSTM unit has an internal state, and this state is updated recurrently at each time step. At the output layer, a softmax activation function is used to produce the estimation of normalized probability values of possible calls coming next in the sequence, P (xi|x1:i−1). According to the chain rule, we can estimate the sequence probability by the following formula:
P (x) = l∏ i=1 P (xi|x1:i−1) (1)
Given normal training system call sequence data, we can train this LSTM-based system call language model using the back-propagation through time (BPTT) algorithm. The training criterion minimizes the cross-entropy loss, which is equivalent to maximizing the likelihood of the system call sequence. A standard RNN often suffers from the vanishing/exploding gradient problem, and when training with BPTT, gradient values tend to blow up or vanish exponentially. This makes it difficult to learn long-term dependency in RNNs (Bengio et al., 1994). LSTM, a well-designed RNN architecture component, is equipped with an explicit memory cell and tends to be more effective to cope with this problem, resulting in numerous successes in recent RNN applications.
(a) language model architecture (b) estimation of sequence probability
embedding layer
hidden layer
output layer
input layer 𝟎 ∙ ∙ ∙ 𝟎 𝟏 𝟎 ∙ ∙ ∙ 𝟎
⋯
𝑃(𝑥1) 𝑃(𝑥2|𝑥1) 𝑃(𝑥3|𝑥1:2) 𝑃(𝑥𝑛|𝑥1:𝑛−1)
𝑥1
fork
𝑥2
setgid
𝑥𝑛−1
ioctl
𝑥𝑛
close
[GO]
Figure 2: System-call language model.
Because typical processes in the system execute a long chain of system calls, the number of system calls required to fully understand the meaning of a system-call sequence is quite large. In addition, the system calls comprising a process are intertwined with each other in a complicated way. The boundaries between system-call sequences are also vague. In this regard, learning long-term dependence is crucial for devising effective intrusion detection systems.
Markov chains and hidden Markov models are widely used probabilistic models that can estimate the probability of the next call given a sequence of previous calls. There has been previous work on using Markov models in intrusion detection systems (Hofmeyr et al., 1998; Hoang et al., 2003; Hu et al., 2009; Yolacan et al., 2014). However, these methods have an inherent limitation in that the probability of the next call is decided by only a finite number of previous calls. Moreover, LSTM can model exponentially more complex functions than Markov models by using continuous space representations. This property alleviates the data sparsity issue that occurs when a large number of previous states are used in Markov models. In short, the advantages of LSTM models compared to Markov models are two folds: the ability to capture long-term dependency and enhanced expressive power.
Given a new query system-call sequence, on the assumption that abnormal call patterns deviate from learned normal patterns, yielding significantly lower probabilities than those of normal call patterns, a sequence with an average negative log-likelihood above a threshold is classified as abnormal, while a sequence with an average negative log-likelihood below the threshold is classified as normal. By changing the threshold value, we can draw a receiver operating characteristic (ROC) curve, which is the most widely used measure to evaluate intrusion detection systems.
Commonly, IDS is evaluated by the ROC curve rather than a single point corresponding to a specific threshold on the curve. Sensitivity to the threshold is shown on the curve. The x-axis of the curve represents false alarm rates, and the y-axis of the curve represents detection rates.1 If the threshold is too low, the IDS is able to detect attacks well, but users would be annoyed due to false alarms. Conversely, if the threshold is too high, false alarm rates becomes lower, but it is easy for IDS to miss attacks. ROC curves closer to (0, 1) means a better classifier (i.e., a better intrusion detection system). The area under curve (AUC) summarizes the ROC curve into a single value in the range [0, 1] (Bradley, 1997).
2.2 ENSEMBLE METHOD TO MINIMIZE FALSE ALARM RATES
Building a ‘strong normal’ model (a model representing system-call sequences with high probabilities of being normal) is challenging because of over-fitting issues. In other words, a lower training loss does not necessarily imply better generalization performance. We can consider two reasons for encountering this issue.
First, it is possible that only normal data were used for training the IDS without any attack data. Learning discriminative features that can separate normal call sequences from abnormal sequences is thus hard without seeing any abnormal sequences beforehand. This is a common obstacle for
1A false alarm rate is the ratio of validation normal data classified as abnormal. A detection rate is the ratio of detected attacks in the real attack data.
almost every anomaly detection problem. In particular, malicious behaviors are frequently hidden and account for only a small part of all the system call sequences.
Second, in theory, we need a huge amount of data to cover all possible normal patterns to train the model satisfactorily. However, doing so is often impossible in a realistic situation because of the diverse and dynamic nature of system call patterns. Gathering live system-call data is harder than generating synthetic system-call data. The generation of normal training data in an off-line setting can create artifacts, because these data are made in fixed conditions for the sake of convenience in data generation. This setting may cause normal patterns to have some bias.
All these situations make it more difficult to choose a good set of hyper-parameters for LSTM architecture. To cope with this challenge, we propose a new ensemble method. Due to the lack of data, different models with different parameters capture slightly different normal patterns. If function f ∈ S∗ 7→ R, which maps a system call sequence to a real value, is given, we can define a thresholding classifier as follows:
Cf (x; θ) = { normal forf(x) ≤ θ; abnormal otherwise.
(2)
Most of the intrusion detection algorithms, including our proposed method, employ a thresholding classifier. For the sake of explanation, we define a term ‘highly normal’ sequence for the classifier Cf as a system call sequence having an extremely low f value so it will be classified as normal even when the threshold θ is sufficiently low to discriminate true abnormals. Highly normal sequences are represented as a flat horizontal line near (1, 1) in the ROC curve. The more the classifier finds highly normal sequences, the longer this line is. Note that a highly normal sequence is closely related to the false alarm rate.
Our goal is to minimize the false alarm rate through the composition of multiple classifiers Cf1 , Cf2 , . . . , Cfm into a single classifier Cf , resulting in accumulated ‘highly normal’ data (here m is the number of classifiers used in the ensemble). This is due to the fact that a low false alarm rate is an important requisite in computer security, especially in intrusion detection systems. Our ensemble method can be represented by a simple formula:
f(x) = m∑ i=1 wiσ(fi(x)− bi). (3)
As activation function σ, we used a leaky ReLU function, namely σ(x) = max(x, 0.001x). Intuitively, the activation function forces potential ‘highly normal’ sequences having f values lower than bi to keep their low f values to the final f value. If we use the regular ReLU function instead, the degree of ‘highly normal’ sequences could not be differentiated. We set the bias term bi to the median of f values of the normal training data. In (3), wi indicates the importance of each classifier fi. Because we do not know the performance of each classifier before evaluation, we set wi to 1/m. Mathematically, this appears to be a degenerated version of a one-layer neural network. The basic philosophy of the ensemble method is that when the classification results from various classifiers are slightly different, we can make a better decision by composing them well. Still, including bad classifiers could degrade the overall performance. By choosing classifiers carefully, we can achieve satisfactory results in practice, as will be shown in Section 3.2.
2.3 BASELINE CLASSIFIERS
Deep neural networks are an excellent representation learning method. We exploit the sequence representation learned from the final state vector of the LSTM layer after feeding all the sequences of calls. For comparison with our main classifier, we use two baseline classifiers that are commonly used for anomaly detection exploiting vectors corresponding to each sequence: k-nearest neighbor (kNN) and k-means clustering (kMC). Examples of previous work for mapping sequences into vectors of fixed-dimensional hand-crafted features include normalized frequency and tf-idf (Liao & Vemuri, 2002; Xie et al., 2014).
Let T be a normal training set, and let lstm(x) denotes a learned representation of call sequence x from the LSTM layer. kNN classifiers search for k nearest neighbors in T of query sequence x
on the embedded space and measure the minimum radius to cover them all. The minimum radius g(x; k) is used to classify query sequence x. Alternatively, we can count the number of vectors within the fixed radius, g(x; r). In this paper, we used the former. Because the computational cost of a kNN classifier is proportional to the size of T , using a kNN classifier would be intolerable when the normal training dataset becomes larger.
g(x; k) = min r s.t. ∑ y∈T [ d(lstm(x), lstm(y)) ≤ r ] ≥ k (4)
g(x; r) = 1− 1 |T | ∑ y∈T [ d(lstm(x), lstm(y)) ≤ r ] (5)
The kMC algorithm partitions T on the new vector space into k clusters G1, G2, . . . , Gk in which each vector belongs to the cluster with the nearest mean so as to minimize the within-cluster sum of squares. They are computed by Lloyd’s algorithm and converge quickly to a local optimum. The minimum distance from each center of clusters µi, h(x; k), is used to classify the new query sequence.
h(x; k) = min i=1,··· ,k d(lstm(x), µi) (6)
The two classifiers Cg and Ch are closely related in that the kMC classifier is equivalent to the 1-nearest neighbor classifier on the set of centers. In both cases of kNN and kMC, we need to choose parameter k empirically, depending on the distribution of vectors. In addition, we need to choose a distance metric on the embedding space; we used the Euclidean distance measure in our experiments.
3 EXPERIMENTAL RESULTS AND DISCUSSION
3.1 DATASETS
Though system call traces themselves might be easy to acquire, collecting or generating a sufficient amount of meaningful traces for the evaluation of intrusion detection systems is a nontrivial task. In order to aid researchers in this regard, the following datasets were made publicly available from prior work: ADFA-LD (Creech & Hu, 2013), KDD98 (Lippmann et al., 2000) and UNM (of New Mexico, 2012). The KDD98 and UNM datasets were released in 1998 and 2004, respectively. Although these two received continued criticism about their applicability to modern systems (Brown et al., 2009; McHugh, 2000; Tan & Maxion, 2003), we include them as the results would show how our model fares against early works in the field, which were mostly evaluated on these datasets. As the ADFALD dataset was generated around 2012 to reflect contemporary systems and attacks, we have done our evaluation mainly on this dataset.
The ADFA-LD dataset was captured on an x86 machine running Ubuntu 11.04 and consists of three groups: normal training traces, normal validation traces, and attack traces. The KDD98 dataset was audited on a Solaris 2.5.1 server. We processed the audit data into system call traces per session. Each session trace was marked as normal or attack depending on the information provided in the accompanied bsm.list file, which is available alongside the dataset. Among the UNM process set, we tested our model with lpr that was collected from SunOS 4.1.4 machines. We merged the live lpr set and the synthetic lpr set. This combined dataset is further categorized into two groups: normal traces and attack traces. To maintain consistency with ADFA-LD, we divided the normal data of KDD98 and UNM into training and validation data in a ratio of 1:5, which is the ratio of the ADFA-LD dataset. The numbers of system-call sequences in each dataset we used are summarized in Table 1.
3.2 PERFORMANCE EVALUATION
We used ADFA-LD and built three independent system-call language models by changing the hyperparameters of the LSTM layer: (1) one layer with 200 cells, (2) one layer with 400 cells, and (3) two layers with 400 cells. We matched the number of cells and the dimension of the embedding
vector. Our parameters were uniformly initialized in [−0.1, 0.1]. For computational efficiency, we adjusted all system-call sequences in a mini-batch to be of similar lengths. We used the Adam optimizer (Kingma & Ba, 2014) for stochastic gradient descent with a learning rate of 0.0001. The normalized gradient was rescaled whenever its norm exceeded 5 (Pascanu et al., 2013), and we used dropout (Srivastava et al., 2014) with probability 0.5. We show the ROC curves obtained from the experiment in Figure 3.
For the two baseline classifiers, we used the Euclidean distance measure. Changing the distance measure to another metric did not perform well on average. In case of kNN, using k = 11 achieved the best performance empirically. For kMC, using k = 1 gave the best performance. Increasing the value of k produced similar but poorer results. We speculate the reason why a single cluster suffices as follows: learned representation vectors of normal training sequence are symmetrically distributed. The kNN classifier Cg and the kMC classifier Ch achieved similar performance. Compared to Liao & Vemuri (2002); Xie et al. (2014), our baseline classifiers easily returned ‘highly normal’ calls. This result was leveraged by the better representation obtained from the proposed system-call language modeling.
As shown in the left plot of Figure 3, three LSTM classifiers performed better than Cg and Ch. We assume that the three LSTM classifiers we trained are strong enough by themselves, and their classification results would be different from each other. By applying ensemble methods, we would expect to improve the performance. The first one was averaging, the second one was voting, and lastly we used our ensemble method as we explained in Section 2.2. The proposed ensemble method gave a better AUC value (0.928) with a large margin than that of the averaging ensemble method (0.890) and the voting ensemble method (0.859). Moreover, the curve obtained from the proposed ensemble method was placed above individual single curves, while other ensemble methods did not show this property.
In the setting of anomaly detection where attack data are unavailable, learning ensemble parameters is infeasible. If we exploit partial attack data, the assumption breaks down and the zero-day attack issue remains. Our ensemble method is appealing in that it performs remarkably well without learning.
To be clear, we applied ensemble methods to three LSTM classifiers learned independently using different hyper-parameters, not with the baseline classifiers, Cg or Ch. Applying ensemble methods to each type of baseline classifier gave unsatisfactory results since changing parameters or initialization did not result in complementary and reasonable classifiers that were essential for ensemble methods. Alternatively, we could do ensemble our LSTM classifiers and baseline classifiers together. However, this would also be a wrong idea because their f values differ in scale. The value of f in our LSTM classifier is an average negative log-likelihood, whereas g and h indicate distances in a continuous space.
According to Creech & Hu (2014), the extreme learning machine (ELM) model, sequence timedelay embedding (STIDE), and the hidden Markov model (HMM) (Forrest et al., 1996; Warrender et al., 1999) achieved about 13%, 23%, and 42% false alarm rates (FAR) for 90% detection rate (DR), respectively. We achieved 16% FAR for 90% DR, which is comparable to the result of ELM and outperforms those of STIDE and HMM. The ROC curves for ELM, HMM, and STIDE can be found, but we could not draw those curves on the same plot with ours because the authors provided no specific details of their results. Creech & Hu (2014) classified ELM as a semantic approach and other two as syntactic approaches which treat each call as a basic unit. To be fair, our proposed method should be compared with those approaches that use system calls only as a basic unit in that we watch the sequence call-by-call. Furthermore, our method is end-to-end while ELM relies on hand-crafted features.
In Creech & Hu (2014), the authors reported that there was significant overhead for training the models mentioned above, and the overhead would inevitably increase for handling larger data. Longer phrases tend to be more informative, but handling them typically requires larger dictionaries. For this reason, Creech & Hu (2014) had to put an empirical upper bound to limit the lengths of phrases, which then might lower the performance of the models to handle various attacks. By contrast, our approach can learn in continuous space semantically meaningful representations of calls, phrases, and sequences of arbitrary lengths. Moreover, our method can relieve the burden of preprocessing (potentially massive) logging data. We expect that incorporating prior knowledge into our model can further boost its performance.
3.3 PORTABILITY EVALUATION
We carried out experiments similar to those presented in Section 3.2 using the KDD98 dataset and the UNM dataset. First, we trained our system-call language model with LSTM having one layer of 200 cells and built our classifier using the normal training traces of the KDD98 dataset. The same model was used to evaluate the UNM dataset to examine the portability of the LSTM models trained with data from a different but similar system. The results of our experiments are represented in Figure 4. For comparison, we display the ROC curve of the UNM dataset by using the model from training the normal traces therein. To examine portability, the system calls in test datasets need to be included or matched to those of training datasets. UNM was generated using an earlier version of OS than that of KDD98, but ADFA-LD was audited on a fairly different OS. This made our experiments with other combinations difficult.
Through a quantitative analysis, for the KDD98 dataset, we earned an almost perfect ROC curve with an AUC value of 0.994 and achieved 2.3% FAR for 100% DR. With the same model, we tested the UNM datset and obtained a ROC curve with an AUC value of 0.969 and 5.5% FAR for 99.8% DR. This result was close to the result earned by using the model trained on normal training traces of the UNM dataset itself, as shown in the right plot of Figure 4.
This result is intriguing because it indicates that system-call language models have a strong portability. In other words, after training one robust and extensive model, the model can then be deployed to other similar host systems. By doing so, we can mitigate the burden of training cost. This paradigm is closely related to the concept of transfer learning, or zero-shot learning. It is well known that neural networks can learn abstract features and that they can be used successfully for unseen data.
3.4 VISUALIZATION OF LEARNED REPRESENTATIONS
It is well-known that neural network based-language models can learn semantically meaningful embeddings to continuous space (Bengio et al., 2003; Mikolov et al., 2013; Cho et al., 2014). We
expected to see a similar characteristic with the proposed system-call language model. The 2D projection of the calls using the embedding matrix W learned from the system-call language model was done by t-SNE (Van der Maaten & Hinton, 2008) and shown in Figure 5. Just as the natural language model, we can expect that calls having similar co-occurrence patterns are positioned in similar locations in the embedded space after training the system call language model. We can clearly see that calls having alike functionality are clustered with each other.
The first obvious cluster would be the read-write call pair and the open-close pair. The calls of each pair were located in close proximity in the space, meaning that our model learned to associate them together. At the same time, the difference between the calls of each pair appears to be almost the same in the space, which in turn would mean our model learned that the relationship of each pair somewhat resembles.
Another notable cluster would be the group of select, pselect6, ppoll, epoll wait and nanosleep. The calls select, pselect6 and ppoll all have nearly identical functions in that they wait for some file descriptors to become ready for some class of I/O operation or for signals. The other two calls also have similar characteristics in that they wait for a certain event or signal as well. This could be interpreted as our model learning that these ‘waiting’ calls share similar characteristics.
Other interesting groups would be: readlink and lstat64 which are calls related to symbolic links; fstatat64 and fstat64 which are calls related to stat calls using file descriptors; pipe and pipe2 which are nearly identical and appear almost as one on the embedding layer. These cases show that our model is capable of learning similar characteristics among the great many system calls.
Similarly to the call representations, we expected that attack sequences with the same type would cluster to each other, and we tried to visualize them. However, for various reasons including the lack of data, we were not able to observe this phenomenon. Taking the fact that detecting abnormal patterns from normal patterns well would be sufficiently hard into consideration, learning representation to separate different abnormal patterns with only seen normal patterns would also be an extremely difficult task.
4 CONCLUSION
Our main contributions for designing intrusion detection systems as described in this paper have two parts: the introduction of a system-call language modeling approach and a new ensemble method. To the best of the authors’ knowledge, our method is the first to introduce the concept of a language model, especially using LSTM, to anomaly-based IDS. The system-call language model can capture the semantic meaning of each call and its relation to other system calls. Moreover, we proposed an innovative and simple ensemble method that can better fit to IDS design by focusing on lowering false alarm rates. We showed its outstanding performance by comparing it with existing state-of-theart methods and demonstrated its robustness and generality by experiments on diverse benchmarks.
As discussed earlier, the proposed method also has excellent portability. In contrast to alternative methods, our proposed method incurs significant smaller training overhead because it does not need to build databases or dictionaries to keep a potentially exponential amount of patterns. Our method is compact and light in that the size of the space required to save parameters is small. The overall training and inference processes are also efficient and fast, as our methods can be implemented using efficient sequential matrix multiplications.
As part of our future work, we are planning to tackle the task of detecting elaborate contemporary attacks including mimicry attacks (Wagner & Soto, 2002; Shu et al., 2015) by more advanced methods. Our proposed method allows us to estimate the likelihood of arbitrary sections in a given system-call sequence, which may be helpful for analyzing the capability of handling mimicry attacks. For instance, it is possible to determine if there exists a sufficiently long section (rather than the whole sequence) with the average log-likelihood below the threshold. In addition, we are considering designing a new framework to build a robust model in on-line settings by collecting large-scale data generated from distributed environments. For optimization of the present work, we would be able to alter the structure of RNNs used in our system-call language model and ensemble algorithm. Finally, we anticipate that a hybrid method that combines signature-based approaches and feature engineering will allow us to create more accurate intrusion detection systems.
ACKNOWLEDGMENTS
This work was supported by BK21 Plus Project in 2016 (Electrical and Computer Engineering, Seoul National University).

The authors propose using an LSTM on a sequence of system calls to perform network intrusion detection (NIDS). The idea of using neural networks (in general) for NIDS is old [1]. The idea of using some sort of NN on top of a sequence of system calls for NIDS is published [2]. The idea of using LSTMs for NIDS is published [2]. The paper in [2] operates on counts of N-grams of system calls, rather than on the raw sequence, but that pre-processing does not seem "heavy" to me.
Overall, the proposed system works as well as other proposed NIDS system, and the paper checks portability (which is good).
But, on the con side, I don't see this paper as adding a lot to the state-of-the-art in NIDS. Nor does is the paper well-matched to ICLR. I didn't learn a lot about representations from this paper: many people have thrown LSTM at sequence problems.
Therefore, I think it's below threshold for ICLR. The authors may wish to submit to a security conference.
References:
1. Debar, Herve, Monique Becker, and Didier Siboni. "A neural network component for an intrusion detection system." Research in Security and Privacy, 1992. Proceedings., 1992 IEEE Computer Society Symposium on. IEEE, 1992.
2. Creech, Gideon, and Jiankun Hu. "A semantic approach to host-based intrusion detection systems using contiguousand discontiguous system call patterns." IEEE Transactions on Computers 63.4 (2014): 807-819.
3. Staudemeyer, Ralf C. "Applying long short-term memory recurrent neural networks to intrusion detection." South African Computer Journal 56.1 (2015).


GENERATIVE PARAGRAPH VECTOR
The recently introduced Paragraph Vector is an efficient method for learning highquality distributed representations for pieces of texts. However, an inherent limitation of Paragraph Vector is lack of ability to infer distributed representations for texts outside of the training set. To tackle this problem, we introduce a Generative Paragraph Vector, which can be viewed as a probabilistic extension of the Distributed Bag of Words version of Paragraph Vector with a complete generative process. With the ability to infer the distributed representations for unseen texts, we can further incorporate text labels into the model and turn it into a supervised version, namely Supervised Generative Paragraph Vector. In this way, we can leverage the labels paired with the texts to guide the representation learning, and employ the learned model for prediction tasks directly. Experiments on five text classification benchmark collections show that both model architectures can yield superior classification performance over the state-of-the-art counterparts.
1 INTRODUCTION
A central problem in many text based applications, e.g., sentiment classification (Pang & Lee, 2008), question answering (Stefanie Tellex & Marton., 2003) and machine translation (I. Sutskever & Le, 2014), is how to capture the essential meaning of a piece of text in a fixed-length vector. Perhaps the most popular fixed-length vector representations for texts is the bag-of-words (or bag-of-ngrams) (Harris, 1954). Besides, probabilistic latent semantic indexing (PLSI) (Hofmann, 1999) and latent Dirichlet allocation (LDA) (Blei & Jordan, 2003) are two widely adopted alternatives.
A recent paradigm in this direction is to use a distributed representation for texts (T. Mikolov & Dean, 2013a). In particular, Le and Mikolov (Quoc Le, 2014; Andrew M.Dai, 2014) show that their method, Paragraph Vector (PV), can capture text semantics in dense vectors and outperform many existing representation models. Although PV is an efficient method for learning high-quality distributed text representations, it suffers a similar problem as PLSI that it provides no model on text vectors: it is unclear how to infer the distributed representations for texts outside of the training set with the learned model (i.e., learned text and word vectors). Such a limitation largely restricts the usage of the PV model, especially in those prediction focused scenarios.
Inspired by the completion and improvement of LDA over PLSI, we first introduce the Generative Paragraph Vector (GPV) with a complete generation process for a corpus. Specifically, GPV can be viewed as a probabilistic extension of the Distributed Bag of Words version of Paragraph Vector (PVDBOW), where the text vector is viewed as a hidden variable sampled from some prior distributions, and the words within the text are then sampled from the softmax distribution given the text and word vectors. With a complete generative process, we are able to infer the distributed representations of new texts based on the learned model. Meanwhile, the prior distribution over text vectors also acts as a regularization factor from the view of optimization, thus can lead to higher-quality text representations.
More importantly, with the ability to infer the distributed representations for unseen texts, we now can directly incorporate labels paired with the texts into the model to guide the representation learning, and turn the model into a supervised version, namely Supervised Generative Paragraph Vector (SGPV). Note that supervision cannot be directly leveraged in the original PV model since it has no
generalization ability on new texts. By learning the SGPV model, we can directly employ SGPV to predict labels for new texts. As we know, when the goal is prediction, fitting a supervised model would be a better choice than learning a general purpose representations of texts in an unsupervised way. We further show that SGPV can be easily extended to accommodate n-grams so that we can take into account word order information, which is important in learning semantics of texts.
We evaluated our proposed models on five text classification benchmark datasets. For the unsupervised GPV, we show that its superiority over the existing counterparts, such as bag-of-words, LDA, PV and FastSent (Felix Hill, 2016). For the SGPV model, we take into comparison both traditional supervised representation models, e.g. MNB (S. Wang, 2012), and a variety of state-of-the-art deep neural models for text classification (Kim, 2014; N. Kalchbrenner, 2014; Socher & Potts, 2013; Irsoy & Cardie, 2014). Again we show that the proposed SGPV can outperform the baseline methods by a substantial margin, demonstrating it is a simple yet effective model.
The rest of the paper is organized as follows. We first review the related work in section 2 and briefly describe PV in section 3. We then introduce the unsupervised generative model GPV and supervised generative model SGPV in section 4 and section 5 respectively. Experimental results are shown in section 6 and conclusions are made in section 7.
2 RELATED WORK
Many text based applications require the text input to be represented as a fixed-length feature vector. The most common fixed-length representation is bag-of-words (BoW) (Harris, 1954). For example, in the popular TF-IDF scheme (Salton & McGill, 1983), each document is represented by tfidf values of a set of selected feature-words. However, the BoW representation often suffers from data sparsity and high dimension. Meanwhile, due to the independent assumption between words, BoW representation has very little sense about the semantics of the words.
To address this shortcoming, several dimensionality reduction methods have been proposed, such as latent semantic indexing (LSI) (S. Deerwester & Harshman, 1990), Probabilistic latent semantic indexing (PLSI) (Hofmann, 1999) and latent Dirichlet allocation (LDA) (Blei & Jordan, 2003). Both PLSI and LDA have a good statistical foundation and proper generative model of the documents, as compared with LSI which relies on a singular value decomposition over the term-document cooccurrence matrix. In PLSI, each word is generated from a single topic, and different words in a document may be generated from different topics. While PLSI makes great effect on probabilistic modeling of documents, it is not clear how to assign probability to a document outside of the training set with the learned model. To address this issue, LDA is proposed by introducing a complete generative process over the documents, and demonstrated as a state-of-the-art document representation method. To further tackle the prediction task, Supervised LDA (David M.Blei, 2007) is developed by jointly modeling the documents and the labels.
Recently, distributed models have been demonstrated as efficient methods to acquire semantic representations of texts. A representative method is Word2Vec (Tomas Mikolov & Dean, 2013b), which can learn meaningful word representations in an unsupervised way from large scale corpus. To represent sentences or documents, a simple approach is then using a weighted average of all the words. A more sophisticated approach is combing the word vectors in an order given by a parse tree (Richard Socher & Ng, 2012). Later, Paragraph Vector (PV) (Quoc Le, 2014) is introduced to directly learn the distributed representations of sentences and documents. There are two variants in PV, namely the Distributed Memory Model of Paragraph Vector (PV-DM) and the Distributed Bag of Words version of Paragraph Vector (PV-DBOW), based on two different model architectures. Although PV is a simple yet effective distributed model on sentences and documents, it suffers a similar problem as PLSI that it provides no model on text vectors: it is unclear how to infer the distributed representations for texts outside of the training set with the learned model.
Besides these unsupervised representation learning methods, there have been many supervised deep models with directly learn sentence or document representations for the prediction tasks. Recursive Neural Network (RecursiveNN) (Richard Socher & Ng, 2012) has been proven to be efficient in terms of constructing sentence representations. Recurrent Neural Network (RNN) (Ilya Sutskever & Hinton, 2011) can be viewed as an extremely deep neural network with weight sharing across time. Convolution Neural Network (CNN) (Kim, 2014) can fairly determine discriminative phrases in a
text with a max-pooling layer. However, these deep models are usually quite complex and thus the training would be time-consuming on large corpus.
3 PARAGRAPH VECTOR
Since our model can be viewed as a probabilistic extension of the PV-DBOW model with a complete generative process, we first briefly review the PV-DBOW model for reference.
In PV-DBOW, each text is mapped to a unique paragraph vector and each word is mapped to a unique word vector in a continuous space. The paragraph vector is used to predict target words randomly sampled from the paragraph as shown in Figure 1. More formally, Let D={d1, . . . ,dN} denote a corpus of N texts, where each text dn = (wn1 , w n 2 , . . . , w n ln
), n ∈ 1, 2, . . . , N is an lnlength word sequence over the word vocabulary V of size M . Each text d ∈ D and each word w ∈ V is associated with a vector ~d ∈ RK and ~w ∈ RK , respectively, where K is the embedding dimensionality. The predictive objective of the PV-DBOW for each word wnl ∈ dn is defined by the softmax function
p(wni |dn) = exp(~wni · ~dn)∑ w′∈V exp(~w ′ · ~dn) (1)
The PV-DBOW model can be efficiently trained using the stochastic gradient descent (Rumelhart & Williams, 1986) with negative sampling (T. Mikolov & Dean, 2013a).
As compared with traditional topic models, e.g. PLSI and LDA, PV-DBOW conveys the following merits. Firstly, PV-DBOW using negative sampling can be interpretated as a matrix factorization over the words-by-texts co-occurrence matrix with shifted-PMI values (Omer Levy & Ramat-Gan, 2015). In this way, more discriminative information (i.e., PMI) can be modeled in PV as compared with the generative topic models which learn over the words-by-texts co-occurrence matrix with raw frequency values. Secondly, PV-DBOW does not have the explicit “topic” layer and allows words automatically clustered according to their co-occurrence patterns during the learning process. In this way, PV-DBOW can potentially learn much finer topics than traditional topic models given the same hidden dimensionality of texts. However, a major problem with PV-DBOW is that it provides no model on text vectors: it is unclear how to infer the distributed representations for unseen texts.
4 GENERATIVE PARAGRAPH VECTOR
In this section, we introduce the GPV model in detail. Overall, GPV is a generative probabilistic model for a corpus. We assume that for each text, a latent paragraph vector is first sampled from some prior distributions, and the words within the text are then generated from the normalized exponential (i.e. softmax) distribution given the paragraph vector and word vectors. In our work, multivariate normal distribution is employed as the prior distribution for paragraph vectors. It could
be replaced by other prior distributions and we will leave this as our future work. The specific generative process is as follows:
For each text dn ∈D, n = 1, 2, . . . , N : (a) Draw paragraph vector ~dn ∼ N (µ,Σ) (b) For each word wni ∈ dn, i = 1, 2, . . . , ln :
Draw word wni ∼ softmax(~dn ·W )i
where W denotes a k ×M word embedding matrix with W∗j = ~wj , and softmax(~dn ·W )i is the softmax function defined the same as in Equation (1). Figure 2 (Left) provides the graphical model of this generative process. Note that GPV differs from PV-DBOW in that the paragraph vector is a hidden variable generated from some prior distribution, which allows us to infer the paragraph vector over future texts given the learned model. Based on the above generative process, the probability of the whole corpus can be written as follows:
p(D)= N∏ n=1 ∫ p(~dn|µ,Σ) ∏ wni ∈dn p(wni |W, ~dn)d~dn
To learn the model, direct maximum likelihood estimation is not tractable due to non-closed form of the integral. We approximate this learning problem by using MAP estimates for ~dn, which can be formulated as follows:
(µ∗,Σ∗,W ∗) = arg max µ,Σ,W
∏ p(d̂n|µ,Σ) ∏ wni ∈dn p(wni |W, d̂n)
where d̂n denotes the MAP estimate of ~dn for dn, (µ∗,Σ∗,W ∗) denotes the optimal solution. Note that for computational simplicity, in this work we fixed µ as a zero vector and Σ as a identity matrix. In this way, all the free parameters to be learned in our model are word embedding matrix W . By taking the logarithm and applying the negative sampling idea to approximate the softmax function, we obtain the final learning problem
L= N∑ n=1 ( −1 2 ||d̂n||2+ ∑ wni ∈dn ( log σ(~wni ·d̂n)+k·Ew′∼Pnw log σ(− ~w′ · d̂n) )) where σ(x) = 1/(1 + exp(−x)), k is the number of “negative” samples, w′ denotes the sampled word and Pnw denotes the distribution of negative word samples. As we can see from the final objective function, the prior distribution over paragraph vectors actually act as a regularization term. From the view of optimization, such regularization term could constrain the learning space and usually produces better paragraph vectors.
For optimization, we use coordinate ascent, which first optimizes the word vectors W while leaving the MAP estimates (d̂) fixed. Then we find the new MAP estimate for each document while leaving the word vectors fixed, and continue this process until convergence. To accelerate the learning, we adopt a similar stochastic learning framework as in PV which iteratively updates W and estimates ~d by randomly sampling text and word pairs.
At prediction time, given a new text, we perform an inference step to compute the paragraph vector for the input text. In this step, we freeze the vector representations of each word, and apply the same MAP estimation process of ~d as in the learning phase. With the inferred paragraph vector of the test text, we can feed it to other prediction models for different applications.
5 SUPERVISED GENERATIVE PARAGRAPH VECTOR
With the ability to infer the distributed representations for unseen texts, we now can incorporate the labels paired with the texts into the model to guide the representation learning, and turn the model into a more powerful supervised version directly towards prediction tasks. Specifically, we introduce an additional label generation process into GPV to accommodate text labels, and obtain the Supervised Generative Paragraph Vector (SGPV) model. Formally, in SGPV, the n-th text dn and the corresponding class label yn ∈ {1, 2, . . . , C} arise from the following generative process:
For each text dn ∈D, n = 1, 2, . . . , N : (a) Draw paragraph vector ~dn ∼ N (µ,Σ) (b) For each word wni ∈ dn, i = 1, 2, . . . , ln :
Draw word wni ∼ softmax(~dn ·W )i (c) Draw label yn|~dn, U, b ∼ softmax(U · ~dn+b)
where U is a C ×K matrix for a dataset with C output labels, and b is a bias term. The graphical model of the above generative process is depicted in Figure 2 (Right). SGPV defines the probability of the whole corpus as follows
p(D)= N∏ n=1 ∫ p(~dn|µ,Σ) ( ∏ wni ∈dn p(wni |W, ~dn) ) p(yn|~dn, U, b)d~dn
We adopt a similar learning process as GPV to estimate the model parameters. Since the SGPV includes the complete generative process of both paragraphs and labels, we can directly leverage it to predict the labels of new texts. Specifically, at prediction time, given all the learned model parameters, we conduct an inference step to infer the paragraph vector as well as the label using MAP estimate over the test text.
The above SGPV may have limited modeling ability on text representation since it mainly relies on uni-grams. As we know, word order information is often critical in capturing the meaning of texts. For example, “machine learning” and “learning machine” are totally different in meaning with the same words. There has been a variety of deep models using complex architectures such as convolution layers or recurrent structures to help capture such order information at the expense of large computational cost.
Here we propose to extend SGPV by introducing an additional generative process for n-grams, so that we can incorporate the word order information into the model and meanwhile keep its simplicity in learning. We name this extension as SGPV-ngram. Here we take the generative process of SGPVbigram as an example.
For each text dn ∈D, n = 1, 2, . . . , N : (a) Draw paragraph vector ~dn ∼ N (µ,Σ) (b) For each word wni ∈ dn, i = 1, 2, . . . , ln :
Draw word wni ∼ softmax(~dn ·W )i
(c) For each bigram gni ∈ dn, i = 1, 2, . . . , sn : Draw bigram gni ∼ softmax(~dn ·G)i
(d) Draw label yn|~dn, U, b ∼ softmax(U · ~dn+b)
where G denotes a K × S bigram embedding matrix with G∗j = ~gj , and S denotes the size of bigram vocabulary. The joint probability over the whole corpus is then defined as
p(D)= N∏ n=1 ∫ p(~dn|µ,Σ) ( ∏ wni ∈dn p(wni |W, ~dn) )( ∏ gni ∈dn p(gni |G, ~dn) ) p(yn|~dn, U, b)d~dn
6 EXPERIMENTS
In this section, we introduce the experimental settings and empirical results on a set of text classification tasks.
6.1 DATASET AND EXPERIMENTAL SETUP
We made use of five publicly available benchmark datasets in comparison.
TREC: The TREC Question Classification dataset (Li & Roth, 2002)1 which consists of 5, 452 train questions and 500 test questions. The goal is to classify a question into 6 different types depending on the answer they seek for.
Subj: Subjectivity dataset (Pang & Lee, 2004) which contains 5, 000 subjective instances and 5, 000 objective instances. The task is to classify a sentence as being subjective or objective.
MR: Movie reviews (Pang & Lee, 2005) 2 with one sentence per review. There are 5, 331 positive sentences and 5, 331 negative sentences. The objective is to classify each review into positive or negative category.
SST-1: Stanford Sentiment Treebank (Socher & Potts, 2013) 3. SST-1 is provided with train/dev/test splits of size 8, 544/1, 101/2, 210. It is a fine-grained classification over five classes: very negative, negative, neutral, positive, and very positive.
SST-2: SST-2 is the same as SST-1 but with neutral reviews removed. We use the standard train/dev/test splits of size 6, 920/872/1, 821 for the binary classification task.
Preprocessing steps were applied to all datasets: words were lowercased, non-English characters and stop words occurrence in the training set are removed. For fair comparison with other published results, we use the default train/test split for TREC, SST-1 and SST-2 datasets. Since explicit split of train/test is not provided by subj and MR datasets, we use 10-fold cross-validation instead.
In our model, text and word vectors are randomly initialized with values uniformly distributed in the range of [-0.5, +0.5]. Following the practice in (Tomas Mikolov & Dean, 2013b) , we set the noise distributions for context and words as pnw(w) ∝ #(w)0.75. We adopt the same linear learning rate strategy where the initial learning rate of our models is 0.025. For unsupervised methods, we use support vector machines (SVM) 4 as the classifier.
6.2 BASELINES
We adopted both unsupervised and supervised methods on text representation as baselines.
6.2.1 UNSUPERVISED BASELINES
Bag-of-word-TFIDF and Bag-of-bigram-TFIDF. In the bag-of-word-TFIDF scheme (Salton & McGill, 1983) , each text is represented as the tf-idf value of chosen feature-words. The bag-of-
1http://cogcomp.cs.illinois.edu/Data/QA/QC/ 2https://www.cs.cornell.edu/people/pabo/movie-review-data/ 3http://nlp.stanford.edu/sentiment/ 4http://www.csie.ntu.edu.tw/˜cjlin/libsvm/
bigram-TFIDF model is constructed by selecting the most frequent unigrams and bigrams from the training subset. We use the vanilla TFIDF in the gensim library5.
LSI (S. Deerwester & Harshman, 1990) and LDA (Blei & Jordan, 2003). LSI maps both texts and words to lower-dimensional representations in a so-called latent semantic space using SVD decomposition. In LDA, each word within a text is modeled as a finite mixture over an underlying set of topics. We use the vanilla LSI and LDA in the gensim library with topic number set as 100.
cBow (Tomas Mikolov & Dean, 2013b). Continuous Bag-Of-Words model. We use average pooling as the global pooling mechanism to compose a sentence vector from a set of word vectors.
PV (Quoc Le, 2014). Paragraph Vector is an unsupervised model to learn distributed representations of words and paragraphs.
FastSent (Felix Hill, 2016). In FastSent, given a simple representation of some sentence in context, the model attempts to predict adjacent sentences.
Note that unlike LDA and GPV, LSI, cBow, and FastSent cannot infer the representations of unseen texts. Therefore, these four models need to fold-in all the test data to learn representations together with training data, which makes it not efficient in practice.
6.2.2 SUPERVISED BASELINES
NBSVM and MNB (S. Wang, 2012). Naive Bayes SVM and Multinomial Naive Bayes with unigrams and bi-grams.
DAN (Mohit Iyyer & III, 2015). Deep averaging network uses average word vectors as the input and applies multiple neural layers to learn text representation under supervision.
CNN-multichannel (Kim, 2014). CNN-multichannel employs convolutional neural network for sentence modeling.
DCNN (N. Kalchbrenner, 2014). DCNN uses a convolutional architecture that replaces wide convolutional layers with dynamic pooling layers.
MV-RNN (Richard Socher & Ng, 2012). Matrix-Vector RNN represents every word and longer phrase in a parse tree as both a vector and a matrix.
DRNN (Irsoy & Cardie, 2014). Deep Recursive Neural Networks is constructed by stacking multiple recursive layers.
Dependency Tree-LSTM (Kai Sheng Tai & Manning, 2015). The Dependency Tree-LSTM based on LSTM structure uses dependency parses of each sentence.
6.3 PERFORMANCE OF GENERATIVE PARAGRAPH VECTOR
We first evaluate the GPV model by comparing with the unsupervised baselines on the TREC, Subj and MR datasets. As shown in table 1, GPV works better than PV over the three tasks. It demonstrates the benefits of introducing a prior distribution (i.e., regularization) over the paragraph vectors. Moreover, GPV can also outperform almost all the baselines on three tasks except Bow-TFIDF and Bigram-TFIDF on the TREC collection. The results show that for unsupervised text representation, bag-of-words representation is quite simple yet powerful which can beat many embedding models. Meanwhile, by using a complete generative process to infer the paragraph vectors, our model can achieve the state-of-the-art performance among the embedding based models.
6.4 PERFORMANCE OF SUPERVISED GENERATIVE PARAGRAPH VECTOR
We compare SGPV model to supervised baselines on all the five classification tasks. Empirical results are shown in Table 2. We can see that SGPV achieves comparable performance against other deep learning models. Note that SGPV is much simpler than these deep models with significantly less parameters and no complex structures. Moreover, deep models with convolutional layers or recurrent structures can potentially capture compositional semantics (e.g., phrases), while SGPV only
5http://radimrehurek.com/gensim/
relies on uni-gram. In this sense, SGPV is quite effective in learning text representation. Meanwhile, if we take Table 1 into consideration, it is not surprising to see that SGPV can consistently outperform GPV on all the three classification tasks. This also demonstrates that it is more effective to directly fit supervised representation models than to learn a general purpose representation in prediction scenarios.
By introducing bi-grams, SGPV-bigram can outperform all the other deep models on four tasks. In particular, the improvements of SGPV-bigram over other baselines are significant on SST-1 and SST-2. These results again demonstrated the effectiveness of our proposed SGPV model on text representations. It also shows the importance of word order information in modeling text semantics.
7 CONCLUSIONS
In this paper, we introduce GPV and SGPV for learning distributed representations for pieces of texts. With a complete generative process, our models are able to infer vector representations as well as labels over unseen texts. Our models keep as simple as PV models, and thus can be efficiently learned over large scale text corpus. Even with such simple structures, both GPV and SGPV can produce state-of-the-art results as compared with existing baselines, especially those complex deep models. For future work, we may consider other probabilistic distributions for both paragraph vectors and word vectors.

This work reframes paragraph vectors from a generative point of view and in so doing, motivates the existing method of inferring paragraph vectors as well as applying a L2 regularizer on the paragraph embeddings. The work also motivates joint learning of a classifier on the paragraph vectors to perform text classification.
The paper has numerous citation issues both in formatting within the text and the formatting of the bibliography, e.g. on some occasions including first names, on others not. I suggest the authors use a software package like BibTex to have a more consistent bibliography. There seems to be little novelty in this work.
The authors claim that there is no proposed method for inferring unseen documents for paragraph vectors. This is untrue. In the original paragraph vector paper, the authors show that to get a new vector, the rest of the model parameters are held fixed and gradient descent is performed on the new paragraph vector. This means the original dataset is not needed when inferring a paragraph vector for new text. This work seems to be essentially doing the same thing when finding the MAP estimate for a new vector. Thus the only contribution from the generative paragraph vector framing is the regularization on the embedding matrix.
The supervised generative paragraph vector amounts to jointly training a linear classifier on the paragraph vectors, while inference for the paragraph vector is unchanged. For the n-gram based approach, the authors should cite Li et al., 2015.
In the experiments, table 1 and 2 are badly formatted with .0 being truncated. The authors also do not state the size of the paragraph vector. Finally the SGPV results are actually worse than that reported in the original paragraph vector paper where SST-1 got 48.7 and SST-2 got 86.3.
Bofang Li, Tao Liu, Xiaoyong Du, Deyuan Zhang, Zhe Zhao, Learning Document Embeddings by Predicting N-grams for Sentiment Classification of Long Movie Reviews, 2015.


Eligibility traces in reinforcement learning are used as a bias-variance trade-off and can often speed up training time by propagating knowledge back over timesteps in a single update. We investigate the use of eligibility traces in combination with recurrent networks in the Atari domain. We illustrate the benefits of both recurrent nets and eligibility traces in some Atari games, and highlight also the importance of the optimization used in the training.
Eligibility traces in reinforcement learning are used as a bias-variance trade-off and can often speed up training time by propagating knowledge back over timesteps in a single update. We investigate the use of eligibility traces in combination with recurrent networks in the Atari domain. We illustrate the benefits of both recurrent nets and eligibility traces in some Atari games, and highlight also the importance of the optimization used in the training.
1 INTRODUCTION
Deep reinforcement learning has had many practical successes in game playing (Mnih et al. (2015),Silver et al. (2016)) and robotics (Levine & Abbeel (2014)). Our interest is in further exploring these algorithms in the context of environments with sparse rewards and partial observability. To this end, we investigate the use of two methods that are known to mitigate these problems: recurrent networks, which provide a form of memory summarizing past experiences, and eligibility traces, which allow information to propagate over multiple time steps. Eligibility traces have been shown empirically to provide faster learning (Sutton & Barto (2017), in preparation) but their use with deep RL has been limited so far (van Seijen & Sutton (2014), Hausknecht & Stone (2015)). We provide experiments in the Atari domain showing that eligibility traces boost the performance of Deep RL. We also demonstrate a surprisingly strong effect of the optimization method on the performance of the recurrent networks.
The paper is structured as follows. In Sec. 2 we provide background and notation needed for the paper. Sec. 3 describes the algorithms we use. In sec. 4 we present and discuss our experimental results. In Sec. 5 we conclude and present avenues for future work.
2 BACKGROUND
A Markov Decision Process (MDP) consists of a tuple 〈S,A, r,P, γ〉, where S is the set of states, A is the set of actions, r : S × A 7→ R is the reward function, P(s′|s, a) is the transition function (giving the next state distribution, conditioned on the current state and action), and γ ∈ [0, 1) is the discount factor. Reinforcement learning (RL) (Sutton & Barto, 1998) is a framework for solving unknown MDPs, which means finding a good (or optimal) way of behaving, also called a policy. RL works by obtaining transitions from the environment and using them, in order to compute a policy that maximizes the expected return, given by E [∑∞ t=0 γ trt ] .
The state-value function for a policy π : S × A → [0, 1], V π(s), is defined as the expected return obtained by starting at state s and picking actions according to π. State-action values Q(s, a) are similar to state values, but conditioned also on the initial action a. A policy can be derived from the Q values by picking always the action with the best estimated value at any state.
Monte Carlo (MC) and Temporal Difference (TD) are two standard methods for updating the value function from data. In MC, an entire trajectory’s return is used as the target value of the current
state.
MC error = ∞∑ i=0 γirt+i − V (st) (1)
In TD, the estimate of the next state’s value is used to correct the current state’s estimate:
TD error = rt + γV (st+1)− V (st) (2)
Q-learning is an RL algorithm that allows an agent to learn by imagining that it will take the best possible action in the following step:
TD error = rt + γmax a′
Q(st+1, a ′)−Q(st, at) (3)
This is an instance of off-policy learning, in which the agent gathers data with an exploratory policy, which randomizes the choice of action, but updates its estimates by constructing targets according to a differnet policy (in this case, the policy that is greedy with respect to the current value estimates.
2.1 ELIGIBILITY TRACES
Eligibility traces are a fundamental reinforcement learning mechanism which allows a trade-off between TD and MC. MC methods suffer from high variance, as many trajectories can be taken from any given state and stochasticity is often present in the MDP. TD suffers from high bias, as it updates values based on its own estimates.
Using eligibility traces allows one to design algorithms that cover the middle-ground between MC and TD. The central notion for these are n-step returns, which provide a way of calculating the target by using the value estimate for the state which occurs n steps in the future (compared to the current state):
R (n) t = n−1∑ i=0 γirt+i + γ nV (st+n). (4)
When n is 1, the results is the TD target, and taking n→∞ yields the MC target. Eligibility traces use a geometric weighting of these n-step returns, where the weight of the k-step return is λ times the weight of the k − 1-step return. Using a λ = 0 reduces to using TD, as all n-steps for n > 1 have a weight of 0. One of the appealing effects of using eligibility traces is that a single update allows states many steps behind a reward signal to receive credit. This propagates knowledge back at a faster rate, allowing for accelerated learning. Especially in environments where rewards are sparse and/or delayed, eligibility traces can help assign credit to past states and actions. Without traces, seeing a sparse reward will only propagate the value back by one step, which in turn needs to be sampled to send the value back a second step, and so on.
Rλt = (1− λ) ∞∑ i=0 λiR (i) t = (1− λ) ∞∑ i=1 λi−1 i−1∑ j=0 γjrj + γ i+1V (st+i) (5)
This way of viewing eligibility traces is called the forward view, as states are looking ahead at the rewards received in the future. The forward view is rarely used, as it requires a state to wait for the future to unfold before calculating an update, and requires memory to store the experience. There is an equivalent view called the backward view, which allows us to calculate updates for every previous state as we take a single action. This requires no memory and lets us perform updates without having to wait for the future. However, this view has had limited success in the neural network setting as it requires using a trace on each neuron of the network, which tend to be dense and heavily used at each step resulting in noisy signals. For this reason, eligibility traces aren’t heavily used when using deep learning, despite their potential benefits.
2.1.1 Q(λ)
Q(λ) is a variant of Q-learning where eligibility traces are used to calculate the TD error. As mentioned previously, the backwards view of traces is traditionally used.
A few versions of Q(λ) exist, but the most used one is Watkins’s Q(λ). As Q-learning is off-policy, the sequence of actions used in the past trajectory used to calculate the trace might be different from the actions that the current policy might take. In that case, one should not be using the trajectory past the point where actions differ. To handle such a case, Watkins’s Q(λ) sets the trace to 0 if the action that the current policy would select is different from the one used in the past.
2.2 DEEP Q-NETWORKS
Mnih et al. (2015) introduced deep Q-networks (DQN), one of the first successful reinforcement learning algorithms that use deep learning for function approximation in a way general enough which is applicable to a variety of environments. Applying it to a set of Atari games, they used a convolutional neural network (CNN) which took as input the last four frames of the game, and output Q-values for each possible action.
Equation 6 shows the DQN cost function, where we are optimizing the θ parameters. The θ− parameters represent frozen Q-value weights which are update at a chosen frequency.
L(st, at|θ) = (rt + γmax a′ Q(st+1, a ′|θ−)−Q(st, at|θ))2 (6)
2.2.1 DEEP RECURRENT Q-NETWORKS
As introduced in Hausknecht & Stone (2015), deep recurrent Q-networks (DRQN) are a modification on DQN, where single frames are passed through a CNN, which generates a feature vector that is then fed to an RNN which finally outputs Q-values. This architecture gives the agent a memory, allowing it to learn long-term temporal effects and handle partial observability, which is the case in many environments. The authors showed that randomly blanking out frames was difficult to overcome for DQN, but that DRQN learned to handle without issue.
To train DRQN, they proposed two variants of experience replay. The first was to sample entire trajectories and run the RNN from end to end. However this is very computationally demanding as some trajectories can be over 10000 steps long. The second alternative was to sample subtrajectories instead of single transitions. This is required as the RNN needs to fill its hidden state and to allow it to understand the temporal aspect of the data.
2.3 OPTIMIZERS
Stochastic gradient descent (SGD) is generally the algorithm used to optimize neural networks. However, some information is lost during the process as past gradients might signal that a weight drastically needs to change, or that it is oscillating, requiring a decrease in learning rate. Adaptive SGD algorithms have been built to use this information.
RMSprop (Tieleman & Hinton (2012)), uses a geometric averaging over gradients squared, and divides the current gradient by its square root. To perform RMSprop, first we calculate the averaging as g = βg + (1− β)∇θ2 and then update the parameters θ ← θ + α ∇θ√
g+ .
DQN (Graves (2013)) introduced a variant of RMSprop where the gradient is instead divided by the standard deviation of the running average. First we calculate the running averages m = βm+ (1− β)∇θ and g = βg + (1 − β)∇θ2, and then update the parameters using θ ← θ + α ∇θ√
g−m2+ . In
the rest of the paper, when mentioning RMSprop, we’ll be referring to this version.
Finally, Kingma & Ba (2014) introduced Adam, which is essentially RMSprop coupled with Nesterov momentum, along with the running averages being corrected for bias. We have a term for the rate of momentum of each of the running averages. To calculate the update with Adam, we start with the updating the averages m = β1m+ (1− β1)∇θ, v = β2v + (1− β2)∇θ2, the correct their biases m̂ = m/(1−βt1), v̂ = v/(1−βt2) and finally calculate the gradient update θ ← θ+α m̂√v̂+ .
3 EXPERIMENTAL SETUP
As explained, the forward view of eligibility traces can be useful, but is computationally demanding in terms of memory and time. One must store all transitions and apply the neural network to each state in the trajectory. By using DRQN, experience replay is already part of the algorithm, which removes the memory requirement of the traces. Then, by training on sub-trajectories of data, the states must be run through the RNN with all state values as the output, which eliminates the computational cost. Finally, all that’s left to use eligibility traces is simply to calculate the weighted sum of the targets, which is very cheap to do.
In this section we analyze the use of eligibility traces when training DRQN and try both RMSprop and Adam as optimizers. We only tested the algorithms on fully observable games as to compare the learning capacities without the unfair advantage of having a memory, as would be the case on partially observable environments.
3.1 ARCHITECTURE
We tested the algorithms on two Atari 2600 games, part of the Arcade Learning Environment (Bellemare et al. (2012)), Pong and Tennis. The architecture used is similar to the one used in Hausknecht & Stone (2015). The frames are converted to gray-scale and re-sized to 84x84. These are then fed to a CNN with the first layer being 32 8x8 filters and a stride of 4, followed by 64 4x4 filters with a stride of 2, then by 64 3x3 filters with a stride of 1. The output of the CNN is then flattened before being fed to a single dense layer of 512 output neurons, which is finally fed to an LSTM (Hochreiter & Schmidhuber (1997)) with 512 cells. We then have a last linear layer that takes the output of the recurrent layer to output the Q-values. All layers before the LSTM are activated using rectified linear units (ReLU).
As mentioned in subsection 2.2.1, we also altered experience replay to sample sub-trajectories. We use backprop through time (BPTT) to train the RNN, but only train on a sub-trajectory of experience. In runtime, the RNN will have had a large sequence of inputs in its hidden state, which can be problematic if always trained with an empty hidden state. Like in Lample & Singh Chaplot (2016), we therefore sample a slightly longer length of trajectory and use the first m states to fill the hidden state. In our experiments, we selected trajectory lengths of 32, where the first 10 states are used as filler and the remaining 22 are used for the traces and TD costs. We used a batch size of 4.
All experiments using eligibility traces use λ = 0.8. Furthermore, we use Watkins’s Q(λ). To limit computation costs of using traces, we cut the trace off once it becomes too small. In our experiments, we choose the limit of 0.01, which allows the traces to affect 21 states ahead (when λ = 0.8). We
calculate the trace for every state in the trajectory, except for a few in the beginning, use to fill in the hidden state of the RNN.
When using RMSprop, we used a momentum of 0.95, an epsilon of 0.01 and a learning rate of 0.00025. When using Adam, we used a momentum of gradients of 0.9, a momentum of squared gradients of 0.999, an epsilon of 0.001 and a learning rate of 0.00025.
Testing phases are consistent across all models, with the score being the average over each game played during 125000 frames. We also use an of 0.05 for action selection.
Choose k as number of trace steps and m as RNN-filler steps Initialize weights θ, experience replay D θ− ← θ s← s0 repeat
Initialize RNN hidden state to 0. repeat
Choose a according to −greedy policy on Q(s, a|θ) Take action a in s, observe s′, r Store s, a, r, s′ in Experience Replay Sample 4 sub-trajectories of m+ k sequential transitions (s, a, r, s′) from D
ŷ = { r s’ is terminal, r + γmax
ā Q(s′, ā|θ−) otherwise
foreach transition sampled do
λt = { λ at = arg maxā(st, ā|θ), 0 otherwise
end for l from 0 to k − 1 do
R̂λt+l = [∑k s=l (∏s i=l λt+i ) R (s−l+1) t+s ] / [∑k s=l (∏s i=l λt+i )] end Perform gradient descent on∂(R̂
λ−Q(s,a|θ))2 ∂θ
Every 10000 steps θ− ← θ s← s′
until s′ is terminal until training complete Algorithm 1: Deep Recurrent Q-Networks with forward view eligibility traces on Atari. The eligibility traces are calculated using the n-step return function R(n)t for time-step t was described in section 2.1.
4 EXPERIMENTAL RESULTS
We describe experiments in two Atari games: Pong and Tennis. We chose Pong because it permits quick experimentation, and Tennis because it is one of the games that has proven difficult in all published results on Atari.
4.1 PONG
First, we tested an RNN model both with λ = 0 and λ = 0.8, trained with RMSprop. Figure 2 shows that the model without a trace (λ = 0) learned at the same rate as DQN, while the model with traces (λ = 0.8) learned substantially faster and with more stability, without exhibiting any epochs with depressed performance. This is probably due to the eligibility traces propagating rewards back by many steps in a single update. In Pong, when the agent hits the ball, it must wait several time-steps before the ball gets either to or past the opponent. Once this happens, the agent must assign the credit of the event back to the time when it hit the ball, and not to the actions performed after the ball had already left. The traces clearly help send this signal back faster.
We then tested the same models but using Adam as the optimizer instead of RMSprop. All models learn much faster with this setting. However, the model with no trace gains significantly more than the model with the trace. Our current intuition is that some hyper-parameters, such as the frozen network’s update frequency, are limiting the rate at which the model can learn. Note also that the DQN model also learns faster with Adam as the optimizer, but remains quite unstable, in comparison with the recurrent net models.
Finally, the results in Table 1 show that both using eligibility traces and Adam provide performance improvements. While training with RMSProp, the model with traces gets to near optimal performance more than twice as quickly as the other models. With Adam, the model learns to be optimal in just 6 epochs.
4.2 TENNIS
The second Atari 2600 game we tested was Tennis. A match consists of only one set, which is won by the player who is the first to win 6 ”games” (as in regular tennis). The score ranges from 24 to -24, given as the difference between the number of balls won by the two players.
As in Pong, we first tried an RNN trained with RMSprop and the standard learning rate of 0.00025, both with and without eligibility traces (using again λ = 0.8 and λ = 0). Figure 3 shows that both RNN models learned to get optimal scores after about 50 epochs. This is in contrast with DQN, which never seems to be able to pass the 0 threshold, with large fluctuations ranging from -24 to 0. After visually inspecting the games played in the testing phase, we noticed that the DQN agent gets stuck in a loop, where it exchanges the ball with the opponent until the timer runs out. In such a case, the agent minimizes the number of points scored against, but never learns to beat the opponent. The score fluctuations depend on how few points the agent allows before entering the loop. We suspect that the agent gets stuck in this policy because the reward for trying to defeat the opponent is delayed, waiting for the ball to reach the opponent and get past it. Furthermore, the experiences of getting a point are relatively sparse. Together, it makes it difficult to propagate the reward back to the action of hitting the ball correctly.
We also notice that both the RNN with and without eligibility traces manage to learn a near-optimal policy without getting stuck in the bad policy. The RNN has the capacity of sending the signal back to past states with BPTT, allowing it to do credit assignment implicitly, which might explain their ability to escape the bad policy. Remarkably, this is the only algorithm that succeeds in getting near-optimal scores on Tennis, out of all variants of DQN (Mnih et al. (2015), Munos et al. (2016), Wang et al. (2015), Mnih et al. (2016), Schaul et al. (2015)), which tend to get stuck in the policy of delaying. The model without traces learned at a faster pace than the one with traces, arriving to a score of 20 in 45 epochs as opposed to 62 for its counterpart. It’s possible that the updates for model with traces were smaller, due to the weighting of target values, indirectly leading to a lower learning rate. We also trained the models with RMSprop and a higher learning rate of 0.001. This led to the model with traces getting to 20 points in just 27 epochs, while the model without traces lost its ability to get optimal scores and never passed the 0 threshold.
We then tried using Adam as the optimizer, with the original learning rate of 0.00025. Both RNN models learned substantially faster than with RMSprop, with the RNN with traces getting to nearoptimal performance in just 13 epochs. With Adam, the gradient for the positive TD is stored in the momentum part of the equation for quite some time. Once in momentum, the gradient is part of many updates, which makes it enough to overtake the safe strategy. We also notice that the model with traces was much more stable than its counterpart. The model without traces fell back to the policy of delaying the game on two occasions, after having learned to beat the opponent. Finally, we trained DQN with Adam, but the model acted the same way as DQN trained with RMSprop.
5 DISCUSSION AND CONCLUSION
In this paper, we analyzed the effects of using eligibility traces and different optimization functions. We showed that eligibility traces can improve and stabilize learning and using Adam can strongly accelerate learning.
As shown in the Pong results, the model using eligibility traces didn’t gain much performance from using Adam. One possible cause is the frozen network. While it has a stabilizing effect in DQN, by stopping policies from drastically changing from a single update, it also stops newly learned values from being propagated back. Double DQN seems to partially go around this issue, allowing
the policy of the next state to change, while keeping the values frozen. In future experiments, we must consider eliminating or increasing the frozen network’s update frequency. It would also be interesting to reduce the size of experience replay, as with increased learning speed, old observations can become too off-policy and barely be used in eligibility traces.

This paper investigates the use of eligibility traces with recurrent DQN agents. As in other recent work on deep RL, the forward view of Sutton and Barto is used to make eligibility traces practical to use with neural networks. Experiments on the Atari games Pong and Tennis show that traces work better than standard Q-learning.
The paper is well written and the use of traces in deep RL is indeed underexplored, but the experiments in the paper are too limited and do not answer the most interesting questions.
As pointed out in the questions, n-step returns have been shown to work better than 1-step returns both in the classical RL literature and more recently with deep networks. [1] shows that using n-step returns in the forward view with neural networks leads to big improvements on both Atari and TORCS. Their n-step Q-learning method also combines returns of different length in expectation, while traces do this explicitly. This paper does not compare traces with n-step returns and simply shows that traces used in the forward view help on two Atari games. This is not a very significant result. It would be much more interesting to see whether traces improve on what is already known to work well with neural networks.
The other claimed contribution of the paper is showing the strong effect of optimization. As with traces, I find it hard to make any conclusions from experiments on two games with fixed hyperparameter settings. This has already been demonstrated with much more thorough experiments in other papers. One could argue that these experiments show that importance of hyperparameter values and not of the optimization algorithm itself. Without tuning the optimization hyperparameters it's hard to claim anything about the relative merits of the methods.
[1] "Asynchronous Methods for Deep Reinforcement Learning", ICML 2016.