Recently several different deep learning architectures have been proposed that take a string of characters as the raw input signal and automatically derive features for text classification. Few studies are available that compare the effectiveness of these approaches for character based text classification with each other. In this paper we perform such an empirical comparison for the important cybersecurity problem of DGA detection: classifying domain names as either benign vs. produced by malware (i.e., by a Domain Generation Algorithm). Training and evaluating on a dataset with 2M domain names shows that there is surprisingly little difference between various convolutional neural network (CNN) and recurrent neural network (RNN) based architectures in terms of accuracy, prompting a preference for the simpler architectures, since they are faster to train and less prone to overfitting.
SUMMARY
This paper addresses the cybersecurity problem of domain generation algorithm (DGA)  detection. A class of malware uses algorithms to automatically generate artificial domain names for various purposes, e.g. to generate large numbers of rendezvous points. DGA detection concerns the (automatic) distinction of actual and artificially generated domain names. In this paper, a basic problem formulation and general solution approach is investigated, namely that of treating the detection as a text classification task and to let domain names arrive to the classifier as strings of characters. A set of five deep learning architectures (both CNNs and RNNs) are compared empirical on the text classification task. A domain name data set with two million instances is used for the experiments. The main conclusion is that the different architectures are almost equally accurate and that this prompts a preference of simpler architectures over more complex architectures, since training time and the likelihood for overfitting can potentially be reduced.
COMMENTS
The introduction is well-written, clear, and concise. It describes the studied real-world problem and clarifies the relevance and challenge involved in solving the problem. The introduction provides a clear overview of deep learning architectures that have already been proposed for solving the problem as well as some architectures that could potentially be used. One suggestion for the introduction is that the authors take some of the description of the domain problem and put it into a separate background section to reduce the text the reader has to consume before arriving at the research problem and proposed solution.
The methods section (Section 2) provides a clear description of each of the five architectures along with brief code listings and details about whether any changes or parameter choices were made for the experiment. In the beginning of the section, it is not clarified why, if a 75 character string is encoded as a 128 byte ASCII sequence, the content has to be stored in a 75 x 128 matrix instead of a vector of size 128. This is clarified later but should perhaps be discussed earlier to allow readers from outside the subarea to grasp the approach.
Section 3 describes the experiment settings, the results, and discusses the learned representations and the possible implications of using either the deep architectures or the “baseline” Random Forest classifier. Perhaps, the authors could elaborate a little bit more on why Random Forests were trained on a completely different set of features than the deep architectures? The data is stated to be randomly divided into training (80%), validation (10%), and testing (10%). How many times is this procedure repeated? (That is, how many experimental runs were averaged or was the experiment run once?).
In summary, this is an interesting and well-written paper on a timely topic. The main conclusion is intuitive. Perhaps the conclusion is even regarded as obvious by some but, in my opinion, the result is important since it was obtained from new, rather extensive experiments on a large data set and through the comparison of several existing (earlier proposed) architectures. Since the main conclusion is that simple models should be prioritised over complex ones (due to that their accuracy is very similar), it would have been interesting to get some brief comments on a simplicity comparison of the candidates at the conclusion.
MINOR COMMENTS
Abstract: “Little studies” -> “Few studies”
Table 1: “approach” -> “approaches”
Figure 1: Use the same y-axis scale for all subplots (if possible) to simplify comparison. Also, try to move Figure 1 so that it appears closer to its inline reference in the text.
Section 3: “based their on popularity” -> “based on their popularity”